Plone

WebLion: A Buildout-Free Plone

Posted 8 days ago

Mentions of WebLion Hosting on the plone-hosting-discussion list recently got my attention. As it turns out, many sysadmins are looking for a way to easily and, most importantly, reproducibly deploy Plone on servers, without being subject to the ... [More] day-to-day shifting of Python package dependencies. As the creator of a hosting service which met those goals but has been left high and dry since Plone became dependent on buildout, I have a lot invested in this problem. I also see buildout-related pain walk through the door of #plone every day. I think we can kill several birds with one stone, simplifying Plone installation for new users and making it amenable to OS-native packaging for professional deployers, without resorting to things like private PyPI mirrors and enormous pinning files, which effectively saddle each person with release manager responsibilities.

Background
First an update on the changes in the Debian Plone scene since WebLion Hosting began....

Debian's Plone and Zope packages have historically been excellent. They tie in with the OS's native upgrade, startup, and configuration mechanisms to ensure a consistent admin experience, and they're extremely flexible, even supporting multiple Zope instances. They're also more reliable and more fail-safe than buildout, about which I go into more detail in WebLion Hosting: Leverating Laziness, Impatience, and Hubris (be sure to click the little Notes tab to show the captions). Unfortunately, the Debian Plone package has recently been abandoned by its maintainer, Fabio Tranchitella, as he felt that fighting the momentum of buildout wasn't worth his time. Zope, meanwhile, continues to be available in Debian Testing.

A Buildout-Free Plone
Despite Fabio's exit, I still need a reliable update mechanism to manage my growing server farm; running buildout in the middle of the night unattended makes my adrenals twitch. So, what I'm working on is a repackaging of Plone that doesn't require buildout and which would therefore be more amenable to OS-native packaging. The simplest form of the distro would ship as a mere tarball, requiring no special expertise to install. The contents would be...

plone/
conf/
var/
plugins/
The plone folder would contain whatever is considered Plone itself: for example, the contents of the current (excellent) Unified Installer. Users shouldn't change anything in this folder. Plone could then be upgraded just by replacing the plone folder with a newer version, and an upgrade could be safely rolled back in the same way, regardless of the network weather. An OS-level package management system would mostly manage the plone folder, and perhaps there would be a helper util, like Debian's dzhandle, to create instances that soft-link to it.

Multiple Zope instances running against a single DB would be handled without instantiating them on disk: Zope has lots of built-in though seldom-used support for passing in such params as port numbers on the command line. I see plone/start taking flags like --instances=4 --port-base=8080 and/or having a plone.conf in the conf folder specifying what to do about multiple instances. Running Plone under a WSGI server (repoze.plone?) and letting it take care of multiprocess stuff is also worthy of consideration.

The conf folder holds all the config files: zope.conf and, if we have one, plone.conf. Users don't have to learn a new buildout syntax for old config directives, and the zope.conf file has lots of self-documenting comments, as of olde.

The var folder is a lot like it is now, full of Data.fs and such.

The plugins folder is where add-ons live. It can support eggs, old-style products, plain-jane packages...whatever. There's no unpredictable build process that has to be run after installing a plugin: just drop it in, and run plone/restart. Repeatability is achieved using tools every sysadmin knows: just tar up the top-level folder, and move it to another machine. For packages that have a lot of dependencies, I could also see a plone/install command that would fetch a plugin and its dependencies into the plugins folder, much like the zenpack --install command in Zenoss. Of course, once you start automatic dependency resolution, you're subject to the [lack of] QA on PyPI. The contents of the plugins folder could be augmented by packages installed in systemwide directories: perhaps by mentioning them in plone.conf.

My goal with the plugins folder is to eliminate the mysterious dependency conflicts and network failures that lead to buildout trashing people's installations. Debian's dependency graph math is reliable because it has a very formal, 3-tier QA process; PyPI is the Wild West by comparison, so buildout can never deliver on its goal of repeatability. It also doesn't hurt that aptitude checks everything before doing anything, making it almost entirely fail-safe.

Implementation
Buildout is a fine developers' tool; it basically makes you your own release manager, especially if you count pinning and such. This makes it suitable (though still not trouble-free) for things like creating and reviewing PLIP bundles, which are full of custom branches and versions of packages. This new distro would probably be easiest to implement as a buildout recipe. I (or whoever ends up maintaining the distro) would run the recipe once for each new Plone version, deal with any network outages or dependency conflicts once and for all, tar up the result, and release it. The result would also be packaged as a deb, an RPM, and whatever else is useful. If successful, I could see this becoming the new contents of the Unified Installer, which could go a long way toward stemming the flow of frightened Installer users into #plone when they re-run buildout for the first time.

Action Items
If this sounds promising to you, there are a few tasks I could use help on. I think most of these could be attacked in parallel:

Dynamic plugins folder. One step is to get the egg-, product-, and package-supporting plugins folder working. I'm about halfway finished with an implementation of this as a Plone product called Abracadabra, which I'll release as a proof of concept.Uninstantiated Zope instances. Figure out how to pass things like port number and socket file location into zopectl via the commandline. Tres Seaver, one of the original Zope crew, assures me the flags should work, and I see them buried about 5 levels deep, but I haven't yet traced them all the way to the surface and made sure they work. Once we do, we won't need each Zope instance to have an on-disk representation, eliminating one big reason for a build process.Helper programs. Decide what commandline management tools would be helpful for packaged installs. Debian's dzhandle program is an excellent start, but it certainly has a few bugs and wouldn't be verbatim steal-able.Think about plone/install. Should we have some kind of tool to automatically download dependencies? How would undo work, for cases where the dependencies are wrong and cause a disaster? How do we avoid totally reimplementing aptitude? This would all be useful without a plone/install command, but it might be easier with.
So, those are my thoughts on simplifying hosting and, as a side effect, Plone's overall distribution experience. Thanks for reading all the way to the bottom! As always, I welcome input, and I especially welcome help, as there's plenty of work to be done. Please mail me if you'd like to help out! I might even be able to put together a sprint (and maybe travel expenses?) in sunny Pennsylvania if folks are interested. We have excellent local microbrews.

Best regards,

Erik Rose / Project Lead / WebLion Hosting / The Pennsylvania State University [Less]

WebLion: WebLion is Hiring

Posted 8 days ago

Penn State University's Plone consultancy and the host of such fine events as Plone Symposium East 2008 and Plone Symposium East 2009: Electric Boogaloo is looking for a top-notch Plonista. We encourage you to apply no matter the nature of your ... [More] Plone-related skills: development, content provider training, theming—whatever excites you. Our business is growing exponentially, and we can use whatever you've got!

Apply here, or ask questions in our IRC channel. [Less]

RedTurtle: 2010 Content Technology Vendor Map

Posted 8 days ago

Webmeisterei: Searching the ideal commenting product for Plone

Posted 8 days ago

We plan to add commenting infrastructure to a portal currently running on plone 3.3.x so I did some research and came up with a (probably uncomplete) list of candidates that will be described below in more detail.

iqpp.plone.commenting (aka ... [More] EasyCommenting)
collective.disqus
quintagroup.plonecomments (aka qPloneComments)
collective.discussionplus
plone.app.discussion

Hope this will kick off a discussion with the product authors / people using them and will help others to decide which package (not) to use.

Summary
I did not look really deep into all of these packages (installed and tested plonecomments and plone.app.discussion) so please feel free to comment and put me into the right direction.

collective.disqus looks promising and has great advantages in the area of social web.
on the other side comments are not as well integrated into plone (eg search) and it’s hard or even impossible? to implement custom notifications or other additional features within your application.

quintagroup.plonecomments is a very mature product with lots of configuration options and notification features.
Editing of comments is possible (maybe there should be a way to turn this evil? feature off).
Comments are compatible with standard plone comments (right?) so they might be migrateable to the new commenting system for plone 4.x (plone.app.discussion).

collective.discussionplus peps up plone’s standard discussion items with workflow while keeping them migrateable to plone.app.discussion later. it’s more leight-weight than plonecomments with the downside of no notifications and less configuration options.

plone.app.discussion is currently in alpha state but offers basic configuration options and a moderation view for batch managing comments. Its much more leightweight than standard plone comment types (plonecomments and collective.dicsussionplus are using). I don’t know how many people are contrubuting currently and i hope timo is still motivated after his plip did not make it into plone 4.0.

I’m not sure if it’s a good idea to use plone.app.discussion in alpha1 state on production servers (comment of the authors welcome). But it seems by using collective.discussionplus or quintagroup.plonecomments (not sure about that) one can migrate all comments with the migration script of plone.app.discussion as soon as there is a more stable release or it even is shipped with plone 4.x

This is just a quick overview, more details for each package follow below. There is a project on coactivate about commenting in plone, too. Founded in 2007 it currently seems to be not very active or completely up to date (eg products like disqus missing) but there are reviews of some other commenting products and a collection of commenting related usecases.

iqpp.plone.commenting
Seems unmaintained (has not been updated on pypi since october 2007), so i did not look into that further.

Martin mentioned in his blogpost on commenting that its comparable to PloneComments and he used it for a big deployment.

collective.disqus
homepage: http://plone.org/products/collective.disqus
demo: http://lichota.pl/blog/

Uses the webservice disqus.com/ so your comments are stored “somewhere in the cloud”.
There is an export for comments though, so you’re able to migrate back to plone comments with some coding.

Seems that you can only set one language in the account settings, so it might won’t work for multilingual sites.

Users can comment using their twitter, facebook, openid,… accounts and choose whether or not to get email notifications on new comments / replies. Comments can be posted on Social Networks with might generate more interest and therefore visits to your site.

Comments are centrally managed using disqus.com.

Since comments are not stored in your plonesite they are not searchable by default.

quintagroup.plonecomments
actively developed (latest release 4.1.1 nov, 2009) and already there for many years.

has it’s own api but pretends to internally use plone’s standard commenting types so migrating to plone.app.discussion might work.

captcha support via quintagroup.plonecaptchas (did not test that)

Configlet for configuring settings (eg activate moderation, or enable anonymous comments) and managing pending comments .

Users with role DiscussionManager can publish comments via publish button below the comments, and can access the configlet and review list (/prefs_recent_comments_form) too.

Notifications for new pending comments are only sent to one single address configured in the control panel.
After Managers have approved the comment users get a notification that their content has been commented and commenter that his comment got published. Commenters get notifications for replies to their comment too (these notifications can be turned on/off in configlet)

Logged in or respectively anonymous users can report spam comments. A configuration option enables an “abuse” button that opens a form users can submit with an optional comment why they consider this comment spam. An email is sent to the notification address.

collective.discussionplus
Martin’s interim solution to enable moderation of standard plone comments.

Uses DCWorkflow, adds indexes to catalog to search for comments of a certain author or the number of comments for a content item. Events are fired for comment creation and removal, so on can hook up email notifications on workflow transitions and/or these events.

In his blogpost on commenting in plone Martin outlined the reasons for not using EasyComment or PloneComments and writing discussionplus.

According to the readme it’s likely not to work with plone4.
However comments should be migrateable to plone.app.discussion since they’re standard plone comments.

plone.app.discussion
Google summer of code project outlined and mentored by martin and mainly developed by timo.

Aimed to be the new default commenting solution for plone 4.0 but the plip9288 has been rejected in october since the framework team considered it not finished enought.

project page: http://plone.org/products/plone.app.discussion/
tracker: http://www.pivotaltracker.com/projects/15135
online demo: http://gsoc.timostollenwerk.net/

For my research I used version 1.0a2 (rev31840) in the development buildout

Configlet to configure if anon comments are allowed, which captcha to use, whether or not to show the commenter’s image and a toggle to switch commenting off globally.

Moderation is configured in the types control panel by setting the workflow of the Comment type.

Captcha support via plone.formwidget.(re)captcha

Migration script for importing existing plone comments is in place (did not test it)

Moderating comments is done using a site action (@@moderate-comments)

Email notifications are not yet available, but planned for 2010 (according to the tracker) [Less]

Izak Burger: Varnish, Zope and Backend Checking

Posted 8 days ago

I sent this explanation about some trouble we had with varnish's backend probing to a client a while ago. The information is useful enough that it should be in a blog post.

David Glick: Recombining ZODB storages

Posted 8 days ago

I recently faced the task of joining back together a Plone site composed of 4 ZODB filestorages that had been (mostly through cavalier naïveté on my part) split asunder some time ago.

Normally I would probably just do a ZEXP export of each ... [More] of the folders that lived in its own mountpoint, then remove the mountpoints and reimport the ZEXP files into the main database. However, that wasn’t going to work in this case because the database included some cross-database references.

Some background: Normally in Zope, mountpoints are the only place where one filestorage references another one, but the ZODB has some support for *any* object to link to any other object in any other database, and this can happen within Zope if you copy an object from one filestorage to another. This is generally bad, since the ZODB’s support for cross-database references is partial — when you pack one filestorage, the garbage collection routine doesn’t know about the cross-database references (unless you use zc.zodbdgc), so an object might get removed even if some other filestorage still refers to it, and you’ll get POSKeyErrors. Also, in ZODB 3.7.x, the code that handles packing doesn’t know about cross-database references, so you’ll get KeyError: ‘m’ or KeyError: ‘n’ while packing.

Well, this is what had happened to my multi-database, and I wanted to keep those cross-database references intact while I merged the site back into one monolithic filestorage. So I ended up adapting the ZEXP export code to:

traverse cross-database references (the standard ZEXP export ignores them and will not include objects in different filestorages from the starting object),
traverse ZODB mountpoints (removing them in the process),
and rewrite all the oids to avoid collisions in the new merged database.

Here is the script I ended up with. If you need to use it, you should:

Edit the final line to pass the object you want to start traversing from, and the filename you want to write the ZEXP dump to.
Run the script using bin/instance run multiexport.py

"""Support for export of multidatabases."""

##############################################################################
#
# Based on the ZODB import/export code.
# Copyright (c) 2009 David Glick.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################

import logging
import cPickle, cStringIO
from ZODB.utils import p64, u64
from ZODB.ExportImport import export_end_marker
from ZODB.DemoStorage import DemoStorage

logger = logging.getLogger('multiexport')

def export_zexp(self, fname):
context = self
f = open(fname, 'wb')
f.write('ZEXP')
for oid, p in flatten_multidatabase(context):
f.writelines((oid, p64(len(p)), p))
f.write(export_end_marker)
f.close()

def flatten_multidatabase(context):
"""Walk a multidatabase and yield rewritten pickles with oids for a single database"""
base_oid = context._p_oid
base_conn = context._p_jar
dbs = base_conn.connections

dummy_storage = DemoStorage()

oids = [(base_conn._db.database_name, base_oid)]
done_oids = {}
# table to keep track of mapping old oids to new oids
ooid_to_oid = {oids[0]: dummy_storage.new_oid()}
while oids:
# loop while references remain to objects we haven't exported yet
(dbname, ooid) = oids.pop(0)
if (dbname, ooid) in done_oids:
continue
done_oids[(dbname, ooid)] = True

db = dbs[dbname]
try:
# get pickle
p, serial = db._storage.load(ooid, db._version)
except:
logger.debug("broken reference for db %s, oid %s", (dbname, repr(ooid)),
exc_info=True)
else:
def persistent_load(ref):
""" Remap a persistent id to a new ID and create a ghost for it.

This is called by the unpickler for each reference found.
"""

# resolve the reference to a database name and oid
if isinstance(ref, tuple):
rdbname, roid = (dbname, ref[0])
elif isinstance(ref, str):
rdbname, roid = (dbname, ref)
else:
try:
ref_type, args = ref
except ValueError:
# weakref
return
else:
if ref_type in ('m', 'n'):
rdbname, roid = (args[0], args[1])
else:
return

# traverse Products.ZODBMountpoint mountpoints to the mounted location
rdb = dbs[rdbname]
p, serial = rdb._storage.load(roid, rdb._version)
klass = p.split()[0]
if 'MountedObject' in klass:
mountpoint = rdb.get(roid)
# get the object with the root as a parent, then unwrap,
# since there's no API to get the unwrapped object
mounted = mountpoint._getOrOpenObject(app).aq_base
rdbname = mounted._p_jar._db.database_name
roid = mounted._p_oid

if roid:
print '%s:%s -> %s:%s' % (dbname, u64(ooid), rdbname, u64(roid))
oids.append((rdbname, roid))

try:
oid = ooid_to_oid[(rdbname, roid)]
except KeyError:
# generate a new oid and associate it with this old db/oid
ooid_to_oid[(rdbname, roid)] = oid = dummy_storage.new_oid()
return Ghost(oid)

# do the repickling dance to rewrite references

pfile = cStringIO.StringIO(p)
unpickler = cPickle.Unpickler(pfile)
unpickler.persistent_load = persistent_load

newp = cStringIO.StringIO()
pickler = cPickle.Pickler(newp, 1)
pickler.persistent_id = persistent_id

pickler.dump(unpickler.load())
pickler.dump(unpickler.load())
p = newp.getvalue()

yield ooid_to_oid[(dbname, ooid)], p

class Ghost(object):
__slots__ = ("oid",)
def __init__(self, oid):
self.oid = oid

def persistent_id(obj):
if isinstance(obj, Ghost):
return obj.oid

export_zexp(app.mysite, '/tmp/mysite.zexp')

Download multiexport.py

I’ve used this script with apparent success, but it has not been extensively tested and your mileage may of course vary. [Less]

Blue Dynamics: Tasty Tasty Soup: cornerstone.soup

Posted 8 days ago

Yesterday I meet Robert to finish work on cornerstone.soup after our cologne alliance partner Sven found an odd bug. We simplified the soup recipe, but now its really yummy. Version 1.0 is released.

cornerstone.soup is a genric storage for ... [More] mass-data in an isolated container. Light-weight records are stored in an IOBTree. A Zope-Tool-Kit catalog in used to index values of interest. cornerstone.soup is no out-of-the-box package. Its addressed to developers needing to solve the problem of storing tiny entities of mass-data, where heavy weight archetypes or dexterity are too much effort and are to slow. I.e if you need a container for non-CMSish content, like votes, data from a poll, orders in a webshop, measuring data, or alike.

Isolated means, for each soup we store all in one persistent local utility. Stored are generic „Records“ which are persistent items with attributes passed in at construction time as keyword arguments. Also the catalog with its indexes is part of the local utility. Items are accessed by query.

We tested the soup with Zope 2.10 and Zope 2.12. It works in a plain Zope. We are using it at the moment in a Plone 3.3

For more information read the documentation at pypi: cornerstone.soup. [Less]

Reinout van Rees: Quotes

Posted 8 days ago

If I see a nice quote, I store it in a textfile somewhere. There've been a
few good ones lately, so here are some. From websites, blogs and mailinglist
messages. No guarantees about correct attribution, though I've tried to write
down ... [More] where I found it.

Culinary mayhem from my brother Maurits:

When dinner is calling, you need a rifle.
Talking about dinner:

It will be like loosing a dragon in the chicken coop.
Project management by Robert Bloch:

The man who smiles when things go wrong has thought of someone to blame it
on.
kcunning has a dire warning:

She will come out like a cat you've dunked in cold bathwater. It will only
be amusing for the person watching it on YouTube later.
Sane programming advice by Brian W. Kernighan:

Debugging is twice as hard as writing the code in the first place. Therefore,
if you write the code as cleverly as possible, you are, by definition, not
smart enough to debug it.
Elizabeth Warren about the financial bump in the road:

The reason banks lost confidence in each other is because they looked at their
own books.
Keep at it, keep at it. Paul Graham:

I've been surprised again and again by just how much more important
persistence is than raw intelligence.
Make noise, embarrass yourself, get famous, be happy. Martijn Faassen:

I'm a natural noise-maker, and I discovered that while as a result of this I
embarrass myself in public on a regular basis, it also means a lot of people
know who I am. That's a good thing.
In Dutch you have a saying "killing a mosquito with a canon". This is a nice
alternative:

It really feels like killing an ant with a sledgehammer.
ESR about the faking of climate data, specifically some of the comments in the
also-released source code:

This isn’t just a smoking gun, it’s a siege cannon with the barrel still
hot.
Chris McDonough about the need for documenting your software:

Document the shit out of it, like it was for your grandma. [Less]

Chris McDonough: My ZCA Thoughts Summary

Posted 9 days ago

Lately there's been some conversation on zope-dev about the future of the "zope.component" package, which is the package that provides "the Zope Component Architecture", more or less. I make some proposals here and take some positions.

Plone.org: Plone domain DNS changes

Posted 9 days ago

Switching DNS servers for Plone domain names.