Writing a Plugin for Storytlr

I recently got interested in the lifestream application Storytlr.

One of the first things I wanted to do was add a plugin for github, as that is a fairly large part of my online life. Unfortunately there is very little documentation, which is understandable for a private project that just went open.

As such I had to work through some code, but all in all it wasn't too tough. The source is very readable, just not set up with any auto-documentation.

So, as a way of giving back, I'm going to walk through the Storytlr plugin system, as I understand it. If you want to play along at home, you can grab the source from my github, and the commit we will be working from is 23136196

First things first, we need to copy off another plugin, that's the easy way to get started. I cloned the RSS one, but they are all pretty similar to start with. Here's what my final file tree looks like, it's pretty similar to the starting tree, just with "Github" instead of "RSS". This would reside at path_to_app/protected/application/plugins/github

.
├── database.sql
├── github.png
├── models
│   ├── GithubItem.php
│   └── GithubModel.php
└── views
    └── scripts
        ├── rss.phtml
        ├── story.phtml
        └── timeline.phtml

So what are each of these and what do we have to do to them? Well, I like to drive my development by my data structures, so let's start with database.sql. This is pretty straightforward SQL, and I'll highlight some key fields after you have a look.

database.sql

DROP TABLE IF EXISTS `github_data`;
SET @saved_cs_client = @@character_set_client;
SET character_set_client = utf8;
CREATE TABLE `github_data` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `source_id` int(10) unsigned NOT NULL,
  `github_id` varchar(255) NOT NULL,
  `title` text NOT NULL,
  `content` text,
  `repository` text,
  `link` varchar(255) NOT NULL,
  `published` varchar(45) NOT NULL,
  PRIMARY KEY USING BTREE (`id`),
  UNIQUE KEY `DUPLICATES` USING BTREE (`source_id`, `github_id`),
  FULLTEXT KEY `SEARCH` (`content`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
SET character_set_client = @saved_cs_client;

So what are our key items to look at here? Well, how about table naming convention? The existing tables are named as their plugin name, all lowercase, then _data. Why get fancy? Let's use the convention.

What else is important? I kept the id and source_id pieces intact. These are actually fairly important, as I learned when I tried to make source_id into a varchar. source_id is actually the key that is tied to the sources table. That seems obvious now, but trust me, it didn't when I was writing this the first time.

Aside from that, there isn't too much to worry about. Just make sure to add something that you can use to identify individual updates on, github_id in this case, and then add your data fields and set your indexes. The naming of the FULLTEXT "SEARCH" key is probably important, though I didn't test it any other way. You'll probably find that all of your plugin tables will look more or less alike.

Now what? Well, let's go ahead and define the model for an item, since we are already thinking about the data types. Here is a partial listing of GithubItem.php, have a look.

class GithubItem extends SourceItem {

  protected $_prefix   = 'github';

  protected $_preamble = 'Github activity: ';

  public function getContent() { return $this->_data['content']; }

  public function getTitle() {
    $title = str_replace(
      $this->_data['repository'],
      '' . $this->_data['repository'] . '',
      html_entity_decode( strip_tags( $this->_data['title'] ) )
    );
    return $title;
  }

  public function getLink() { return $this->_data['link']; }

  public function getType() { return SourceItem::LINK_TYPE; }

  public function getBackup() {
    $item = array();
    $item['SourceID'] = $this->_data['source_id'];
    $item['Title'] = $this->_data['title'];
    $item['Content'] = $this->_data['content'];
    $item['Repository'] = $this->_data['repository'];
    $item['Link'] = $this->_data['link'];
    $item['Published'] = $this->_data['published'];
    return $item;
  }

}

This is a very stripped down implementation. All of your data is available to the views through the inherited SourceItem::toArray method, so you really only need to override a few methods and put in any mangle logic, as I did in the getTitle method.

One other method I was sure to override was getType. The types are enumerated in /protected/application/admin/models/SourceItem.php, and are as follows:

  const IMAGE_TYPE  = 'image';

  const AUDIO_TYPE  = 'audio';

  const VIDEO_TYPE  = 'video';

  const STATUS_TYPE   = 'status';

  const BLOG_TYPE   = 'blog';

  const LINK_TYPE   = 'link';

  const OTHER_TYPE  = 'other';

  const STORY_TYPE  = 'story';

Let's move on to the views. These are all very similar, and are only rendered in different, well, views. I'll go over timeline.phtml, but if you copy the others you should be able to piece it together in a jiffy.

  
item->getTitle(); ?>
  
item->getContent(); ?>

Pretty brutal, huh? You just get your title (mangled by GithubItem::getTitle in this case) and get your content, and you are as good as done.

Finally, I'm going to address the engine that drives all of this, GithubModel.php. A lot of this is just editing boilerplate, so let's start with that. Comments are added in-line, but aren't in the git source.

class GithubModel extends SourceModel {
  // What is the table named?
  protected $_name   = 'github_data';
  // What is the plugin directory named?
  protected $_prefix = 'github';
  // What fields are searchable? This is comma delimited, i.e. "content, title"
  protected $_search  = 'content';
  // What is a format string for update tweets?
  protected $_update_tweet = "Did %d things at github.com on my lifestream %s";
  // What is the service name shown on the backend/widgets
  public function getServiceName() {
    return "Github";
  }
  // ?
  public function isStoryElement() {
    return true;
  }
  // What is the URL for the widget links?
  public function getServiceURL() {
    return 'http://github.com/' . $this->getProperty('username');
  }
  // Brief description for the admin interface
  public function getServiceDescription() {
    return "Github is social coding.";
  }
  // What is the name on the account (for the admin interface mostly)
  public function getAccountName() {
    if ($name = $this->getProperty('username')) {
      return $name;
    }
    else {
      return false;
    }
  }
  // What is the title of the account (for the front end)
  public function getTitle() {
    return $this->getServiceName();
  }
  // The initial data import function
  public function importData() {
    $items = $this->updateData();
    $this->setImported( true );
    return $items;
  }

Okay, now we are ready to dig into the meaty part, the data update functions. This is split into two parts, updateData and processItems. Here is a (hopefully) self-explanatory updateData implementation, more or less cloned from existing plugins.

  public function updateData() {
    $url  = 'http://github.com/' . $this->getProperty('username') . '.atom';

    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_HEADER, false);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_USERAGENT,'Storytlr/1.0');

    $response = curl_exec($curl);
    $http_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);
    curl_close ($curl);

    if ($http_code != 200) {
      throw new Stuffpress_Exception( "Github returned http status $http_code for url: $url", $http_code );
    }

    if (!($items = simplexml_load_string($response))) {
      throw new Stuffpress_Exception( "Github did not return any result", 0 );
    }

    if ( count( $items->entry ) == 0 ) { return; }

    $items = $this->processItems($items->entry);
    $this->markUpdated();
    return $items;
  }

Pretty easy, right? Just acquire the content (any way you wish, this one uses cURL), parse it out into an array and pass it on. If all goes well in processItems, then mark it as a successful update with SourceModel::markUpdated. Let's see what processItems does.

  private function processItems($items) {
    $result = array();
    foreach ($items as $item) {
      $data = array();
      $data['title'] = $item->title;
      $data['repository'] = substr( $item->title, strrpos( $item->title, ' ' ) + 1 );
      $data['published'] = strtotime( $item->published );
      $data['content'] = $item->content;
      $data['link'] = $item->link['href'];
      $data['github_id'] = $item->id;
      $id = $this->addItem( $data, $data['published'], SourceItem::LINK_TYPE, array( $data['repository'] ), false, false, $data['title'] );
      if ($id) $result[] = $id;
    }
    return $result;
  }

Again, pretty simple. We take the passed in items, mangle them as needed and stuff them into a data array for storage. About the only unknown there is the addItem call, which is in the SourceModel class. Let's take a look at the first part of that to understand our parameters.

/protected/application/admin/models/SourceModel.php

  public function addItem($data, $timestamp, $type, $tags=false, $location=false, $hidden=false, $title=false) {
    $data['source_id']   = $this->_source['id'];
    $columns         = array();
    $keys            = array();
    $timestamp       = ($timestamp>=0) ? $timestamp : 0;

    foreach($data as $k => $v) {
      unset($data[$k]);
      if (!$v) continue;
      $columns[] = "$k";
      $keys[] = ":$k";
      $data[":$k"] = "$v";
    }

    $sql = "INSERT IGNORE INTO {$this->_name} (".implode(',', $columns).") "
       . "VALUES(".implode(',', $keys).")";

The most important thing for us to note are the names of the arguments: $data, $timestamp, $type, $tags, $location, $hidden, $title. These are self-explanatory and help us understand why the existing plugins pass what they do. Some other pieces to note is the override of source_id on line 169, and how it builds the query from your $data arguments, on lines 174-183. Naming matters!

So, now we are back to GithubModel, and we just have a few more methods to go. What remains below are the form generators and processing for the admin interface. Github only needs one piece of information to work, the username, so that's all we are asking for below.

GithubModel.php

  public function getConfigForm($populate=false) {
    $form = new Stuffpress_Form();

    // Add the username element
    $element = $form->createElement('text', 'username', array('label' => 'Username', 'decorators' => $form->elementDecorators));
    $element->setRequired(true);
    $form->addElement($element);

    // Populate
    if($populate) {
      $values  = $this->getProperties();
      $form->populate($values);
    }

    return $form;
  }

  public function processConfigForm($form) {
    $values = $form->getValues();
    $update  = false;

    if($values['username'] != $this->getProperty('username')) {
      $this->_properties->setProperty('username',   $values['username']);
      $update = true;
    }

    return $update;
  }
}

getConfigForm creates the form fields using the Stuffpress_Form class, which is described in /protected/library/Stuffpress/Form.php. Github only has the one element, so it is added and set to required, then we are done.

processConfigForm is similarly simple, we get the value of our field, then make sure it is valid. If it is, we save it into our model's properties, which is a SourcesProperties class, which is in turn a Stuffpress_Db_Properties class. Essentially think of it as a persistent key/value store. Or don't think about it at all, just use it.

At this point you should have a working plugin!

Debugging Storytlr can be tough sometimes, so make sure your config file has debug = 1, and keep an eye on /protected/logs/.

If you have any questions, comments or corrections, please let me know!