Utah Open Source Planet

Some of you might have noticed that the Shoutcast plugin hasn't worked for a while, and wasn't fixed with the 0.2.14 update. This is because they changed the way that their site looks, and so the plugin wasn't able to parse it.

A quick look at our forums showed that someone had figured out how to fix it (at least until they change the format again), and so I've made the changes and released version 0.4.8 of the plugin (thanks JDorfler).

Exaile 0.2.14 and 0.3 alpha have been released!

Go to the Exaile homepage for more information.

A nice fellow emailed and asked me if I could post a monthly (possibly weekly) roundup on Exaile 0.3 so that people can follow what's going on. It sounded like a good idea to me, so here's the first one.

We've pretty much ironed out the storage details, and gotten playback/playlists figured out. What we have now is a fully functioning media player. The things that are still missing are:

• Preferences dialog of some sort. We haven't even started on this, however, I feel that the one in 0.2 was pretty well written and could probably be used.
• Cover art manager
• Lyrics manager
• Device manager plugins (for things like Ipods, other mp3 devices)
• Visualizations
• Rating system

Exaile is currently undergoing an almost complete rewrite. Why? Because the codebase could be better. Recently, thanks to Aren Olson, the 0.3 branch has been taking off. Already the code is a lot better, and will be a lot easier to add things to in the future.

As of this writing, you can create a collection, based on different libraries (directories). For each one of these directories, you can set a rescan interval, and/or have the library be watched by pyinotify for changes. You can create playlists, smart playlists, play these playlists in order or shuffle and/or on repeat. I am currently using it as my default player.

However: none of this can be done using a gui yet. This can all be done easily by using Exaile 0.3's pretty good looking internal API. Here is an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/usr/bin/env python

import exaile as ex
from xl.playlists import Playlist
from xl.collection import Library

ex.main()
exaile = ex.exaile

# scan my music directory
exaile.collection.add_library(Library("/home/synic/media/music"))
exaile.rescan_libraries()

# Create a playlist with the artist Thievery Corporation, but exclude anything
# from the Versions album

pl = Playlist()
tracks = exaile.collection.search('artist=="Thievery Corporation" NOT '
    'album="Versions"')

pl.add_tracks(tracks)
pl.toggle_random()  # I usually play my music on shuffle :)

exaile.queue.set_current_playlist(pl)
exaile.queue.next() # start playback of the playlist

# ok, let's quit.  This will save the collection to the database, settings,
# etc.
exaile.quit()

A few versions ago, Exaile had Last.FM streaming support using LastFMSource - a pygst plugin created by Philippe Normand of Elisa. It worked... sort of. Every other time you tried to connect to LastFM station, Gstreamer would lock up entirely, taking Exaile out with it. Not being able to fix this problem, it was eventually removed from Exaile entirely.

Enter: LastFM Proxy. This is a program written in python created to connect to LastFM and start streaming the music to a proxy that you can connect to on your local machine using any music player that supports HTTP streaming.

After a bit of hacking (and really, this is some seriously hackish stuff), I've created a plugin for Exaile called "LastFM Radio" that (mostly) seemlessly integrates LastFMProxy into Exaile. To the user, it appears to just be a Radio Panel plugin like the current "Shoutcast Radio" plugin. The user just clicks on the station they want to listen to, and it starts playing. They can "Skip", "Ban", or "Love" tracks just like in the LastFM native player.

It still needs some work, but overall, I'm pretty pleased with how well it works. Give it a try!

Note: You must be using the latest bzr version of Exaile for this plugin to work. You can get instructions on doing that from Exaile's downloads page.

A nice fellow emailed and asked me if I could post a monthly (possibly weekly) roundup on Exaile 0.3 so that people can follow what's going on. It sounded like a good idea to me, so here's the first one.

We've pretty much ironed out the storage details, and gotten playback/playlists figured out. What we have now is a fully functioning media player. The things that are still missing are:

• Preferences dialog of some sort. We haven't even started on this, however, I feel that the one in 0.2 was pretty well written and could probably be used.
• Cover art manager
• Lyrics manager
• Device manager plugins (for things like Ipods, other mp3 devices)
• Visualizations
• Rating system

Exaile is currently undergoing an almost complete rewrite. Why? Because the codebase could be better. Recently, thanks to Aren Olson, the 0.3 branch has been taking off. Already the code is a lot better, and will be a lot easier to add things to in the future.

As of this writing, you can create a collection, based on different libraries (directories). For each one of these directories, you can set a rescan interval, and/or have the library be watched by pyinotify for changes. You can create playlists, smart playlists, play these playlists in order or shuffle and/or on repeat. I am currently using it as my default player.

However: none of this can be done using a gui yet. This can all be done easily by using Exaile 0.3's pretty good looking internal API. Here is an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/usr/bin/env python

import exaile as ex
from xl.playlists import Playlist
from xl.collection import Library

ex.main()
exaile = ex.exaile

# scan my music directory
exaile.collection.add_library(Library("/home/synic/media/music"))
exaile.rescan_libraries()

# Create a playlist with the artist Thievery Corporation, but exclude anything
# from the Versions album

pl = Playlist()
tracks = exaile.collection.search('artist=="Thievery Corporation" NOT '
    'album="Versions"')

pl.add_tracks(tracks)
pl.toggle_random()  # I usually play my music on shuffle :)

exaile.queue.set_current_playlist(pl)
exaile.queue.next() # start playback of the playlist

# ok, let's quit.  This will save the collection to the database, settings,
# etc.
exaile.quit()

A few versions ago, Exaile had Last.FM streaming support using LastFMSource - a pygst plugin created by Philippe Normand of Elisa. It worked... sort of. Every other time you tried to connect to LastFM station, Gstreamer would lock up entirely, taking Exaile out with it. Not being able to fix this problem, it was eventually removed from Exaile entirely.

Enter: LastFM Proxy. This is a program written in python created to connect to LastFM and start streaming the music to a proxy that you can connect to on your local machine using any music player that supports HTTP streaming.

After a bit of hacking (and really, this is some seriously hackish stuff), I've created a plugin for Exaile called "LastFM Radio" that (mostly) seemlessly integrates LastFMProxy into Exaile. To the user, it appears to just be a Radio Panel plugin like the current "Shoutcast Radio" plugin. The user just clicks on the station they want to listen to, and it starts playing. They can "Skip", "Ban", or "Love" tracks just like in the LastFM native player.

It still needs some work, but overall, I'm pretty pleased with how well it works. Give it a try!

Note: You must be using the latest bzr version of Exaile for this plugin to work. You can get instructions on doing that from Exaile's downloads page.

I’ll admit. I didn’t think this really happened to sites as small as exaile.org, where they are running Linux and not too many services, but I guess that type of thinking is what leads to these types of things.

After it happened I began reading blog posts and other material regarding rootkits and security. I was surprised, and rather distraught to find out that this type of thing happens more than you might think, and that in most cases, the owner of the machine in question doesn’t even know that it’s happened. I don’t know about you, but that worries me to death. If that’s the case, you might be wondering how I found out?

First off, ssh starts acting weird. Instead of the usual:

that you get when you’re trying to connect to a host you’ve never connected to before, I was getting some error message regarding “public keys” that I’ve never seen before, and even weirder, I was getting it when trying to connect to hosts I had already accepted the public key for. I did some manpage reading, and wondered if I had inadvertently switched my preferred authentication type when I was playing around with WebShell (who knows, maybe webshell is the cause of all my problems). The error message I was getting made me thing I had somehow switched to publickey authentication. So, I tried passing the -o PreferredAuthentications=password,keyboard-interactive. I got an error saying “-o PreferredAuthentications is an invalid option”.

Now, I’m no SSH expert, but I had used that option before, and it mentions is specifically in the manpage. By this point, I think I’ve totally broken something. Again, I know I’m not an SSH expert, so I take the Windows approach. I decide that I’m going to just remove ~/.ssh and reinstall openssh-client. apt-get appears to remove everything correctly, but upon reinstall, it claims it can’t overwrite /usr/bin/ssh because the “Operation is not permitted”. What? I’m root. I should be able to remove whatever I want. I jump into #openssh on freenode. I ask them if there is some sort of built in block for security regarding this.

After fiddling around and with some suggestions from a user in that channel, I find out that, for whatever reason, /usr/bin/ssh has been giving the ext2 attributes of “+uia”. These attributes are set using the chattr command. From the chattr manpage:

• +a: A file with the ‘a’ attribute set can only be open in append mode for writing. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.
• +i: A file with the ‘i’ attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file and no data can be written to the file. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.
• +u: When a file with the ‘u’ attribute set is deleted, its contents are saved. This allows the user to ask for its undeletion.

Ok…. great. I had seen the +i attribute before and knew about chattr from my gentoo days. +i is used on the ext3 journal file. No big deal, I figure this is something debian has done for extra security. I unset the attributes with chattr -iua. I try to reinstall openssh-client again. Same thing happens, only with ssh-add. I go to /usr/bin and run chattr -iua ssh-*. The reinstall goes well this time. Ssh however, is still acting weird. I’m getting a little weirded out now. I type which ssh. What does it return? /usr/local/bin/ssh. WHAT THE HELL?!!? Now I know I’ve been hacked, there’s no reason anything would be in /usr/local/anything on this machine. I check /usr/local/bin and notice there’s a myriad of ssh-related commands.

I start scouring the logs. I see things like this:

I’m freaking. Why is root suing to nobody? (I later find out that this is the cronjob that runs daily for the find utility, and it’s totally normal). I check other files in /usr/bin with lsattr. Most of them are set +iua. I keep looking around… using ps aux and looking for processes out of the ordinary. I look around in my web directories… and in http://www.exaile.org/files, I find a directory that’s just named “    ”. That’s right, four spaces. Inside is a directory called “www.irs.gov”, and inside that…. what do I find? A couple of html files and a php script designed to convince whoever that they are on the irs.gov website, and that they are entitled to a sum of over $100 as a return, so long as they fill out their credit card information. A quick look at the php script shows that the information is just logging to a file. Nothing is in the file… yet.

I keep looking around… at this point, I’m sued to root. Suddenly, I see a line that says “Killed”, and I’m back at my user prompt. Someone had killed my bash session. The who command showed only me. I su back to root. Killed again, and then once more, and I’m totally kicked off the server. I try to log back in, and my password doesn’t work. You can imagine how I felt. I wonder to myself “can I log into my webhosting companies support panel and shut the server down?” I try sshing as root, and am able to get in with the root password (I know, I know, I shouldn’t have root logins enabled). I quickly type ‘halt’.

Worst feeling ever. I’m not worried about exaile.org or any of my other opensource websites. I’m kind of a backup nazi… but it’s the other sites I’m hosting for my friends and their companies that I’m worried about. They can’t access their websites, they can’t get their mail.

How does one recover from this? I found out that my hosting company, for $70 (which I didn’t have at the time), will set me up an entirely fresh install of Debian on a new drive with my old drive as a slave. It took me a couple of days to get the money in my account to be able to do so. During this time is when I’m doing my research.

I got my hands on a couple rootkit tarballs. I was able to read the source of them, and found out, that at least for shv5, modifying chattr (by renaming it and replacing it with a script) would render the kit totally useless. Probably other things like renaming tar and etc would stop a script kiddie.

HOWEVER, I realize that these are just little steps (read: hacks) in securing a website. I know that it starts with having a secure setup in the first place… but again, I’m not an expert on this either. So how does a n00b like me secure their piddly servers? I really don’t know. What I do know is that I was being very lax in the first place. I was running proftpd, and I didn’t need to be. I was sending ftp passwords to people on irc via message, though in plain text. My web directory permissions were often just “777” to make things easier when I couldn’t figure out why something wasn’t working. I had heard that the imap and pop3 server (courier) I was using was insecure, but I really wasn’t doing anything about it, and I wasn’t keeping up on my security updates. I was running a private IRC server that was accessible publicly. I’m running PHP... I’ve always heard that’s a bad idea. Probably a bunch of other things.

I’ve got my server back up, and most of the above problems are corrected… all but courier, which is still running, and PHP, which I’m still using. I’ve got courier limited via iptables to certain IPs because I really don’t know what else to do. All my users’ current email depends on courier.

I know that I’m lucky in this case. If the attacker had been anything more than a 13 year old running scripts, I’d probably be hosting some crazy phishing site and a number of IRC bots right now without knowing it. I’ve still got the old drive accessible as a slave, but I don’t know what to do with it (is there a way to find out how the attacker got in?)

Anyway. That’s my story for the day. If any of you have any suggestions, I’d be glad to hear them.

I’m tired. No vim tip for today… other than this: Scrolling is much faster in gVIM. Just a random note.

Exaile now runs on Windows, and runs quite well. Thanks to the GStreamer and Songbird people who recently ported GStreamer to Windows.

Functionally this is equivalent to the patched version in ticket #200, only using GStreamer instead of Windows Media Player.

As ticket #200 shows, it really doesn’t take much to make Exaile run on Windows. Most of it was already cross-platform, largely thanks to the insistence on using os.sep and os.path.join.

It’s as stable and fast as in Linux, and the majority of features work perfectly. Memory consumption is rather high, but then it’s high even in Linux.

Encoding errors abound. Don’t expect to be able to import most of your library.

Anything to do with D-Bus has been chopped off and won’t work. In other words, Exaile doesn’t depend on D-Bus anymore; but that’s a hack, not a feature. I’m still hoping for D-Bus and dbus-python on Windows so I can remove the hack.

Things go wrong when Exaile reloads open playlists from the previous session. From the symptoms, I’m guessing that Exaile thinks those tracks are streams.

How

The changes I made are set for Exaile 0.2.11. For now they’re only available in trunk.

The requirements are basically identical to the ones mentioned in Exaile’s Website, except that you don’t get to apt-get them.

• Python
• GTK+ and libglade for Windows
• PyGTK and its dependencies
• GStreamer, its plugins, and its Python bindings
• Mutagen
• For Python < 2.5, pysqlite
• For Python < 2.5, ElementTree

Right now binary GStreamer Python bindings are only available in the CVS packages, and they’re compiled for Python 2.4 so you’ll need the 2.4 version of everything.

I’m pleased to announce that Exaile gained a new developer last week – Aren Olson. Aren has been contributing here and there for a while now, and as of last week with his new plugin, daap-share.py, I asked him if he wanted commit access. What is DAAP you might ask?

Imagine that while at home, you could load up Exaile and connect to your collection running on your machine at work, search through it just like you search through your local collection, drag some tracks over to your playlist and listen away.

With Aren’s plugin you can, and here’s a little how-to on getting it set up under Ubuntu (Feisty Fawn).

DAAP is the protocol that Apple uses in iTunes to allow different users of iTunes on a local network to share their music with each other.

Aren has created a Feisty repository for Exaile’s svn (which also contains Tangerine [a DAAP server]), which he keeps pretty up to date, so first things first, you’re going to want to add the following lines to your /etc/apt/sources.list:

and then run the following:

Start up Exaile by going to Applications->Sound & Video->Exaile. If this your first time running Exaile, you will be prompted to add some directories to your collection. Do this, and wait for it to complete scanning.

Open up a terminal, and type tangerine-properties. Check the “Enable music sharing” box, the “Find in” radio butt, and choose Exaile from the dropdown. This will start Tangerine, which will begin serving up music from Exaile’s library.

In Exaile, go to Tools->Plugins and select the “Available Plugins” tab. When the list finishes loading (which can take a minute), select the checkbox next to the “Music Sharing” plugin, and click the “Install/Upgrade” button. Close the plugin manager.

You will now see a new tab on the left called “Network”. In the dropdown at the top, you should see your local tangerine share (which will be an IP and a port number, something like 66.79.34.24:52106), and then “Custom Location”. If you are working in an office with people who are running iTunes, you might also see their shares. You can connect to any of these, so long as you have the password, and browse them like they were your own collection.

Now, if you write down that IP and port for your local share, when you get home from your office, you can follow this guide again to get Tangerine and Exaile up and running, choose “Custom Location” from the dropdown in network, and type the IP and port of your work share. AWESOME.

Note, this assumes that you don’t have a firewall at work… you may need to forward a port or something, but I’ll leave that part to you. The hard part is done…. thanks Aren!

Exaile 0.2.10 is well on it’s way.

All plugins have been moved out of the regular Exaile repository, and into their own. The script located here reads information directly from Exaile’s trac source browser to keep up-to-date plugin list (trac watches our svn repository), even if we’ve just committed changes.

This will come to play in the new plugin manager, which I’ve started to work on. When the user opens the plugin manager, they will see an available plugins tab. When they click on that tab, an updated plugin list will be downloaded from the plugin list script running on Exaile.org,

Currently, the user can install plugins from here. Eventually, they will also be able to remove and upgrade plugins from here.

Also, the pluggable radio system is done, and the shoutcast plugin has been written. Everything works as it used to, but new plugins will be able to be written to get radio stations from other stream directories.

Exaile now runs on Windows, and runs quite well. Thanks to the GStreamer and Songbird people who recently ported GStreamer to Windows.

Functionally this is equivalent to the patched version in ticket #200, only using GStreamer instead of Windows Media Player.

As ticket #200 shows, it really doesn’t take much to make Exaile run on Windows. Most of it was already cross-platform, largely thanks to the insistence on using os.sep and os.path.join.

How good

It’s as stable and fast as in Linux, and the majority of features work perfectly. Memory consumption is rather high, but then it’s high even in Linux.

How bad

Encoding errors abound. Don’t expect to be able to import most of your library.

Anything to do with D-Bus has been chopped off and won’t work. In other words, Exaile doesn’t depend on D-Bus anymore; but that’s a hack, not a feature. I’m still hoping for D-Bus and dbus-python on Windows so I can remove the hack.

Things go wrong when Exaile reloads open playlists from the previous session. From the symptoms, I’m guessing that Exaile thinks those tracks are streams.

How it looks like

How

The changes I made are set for Exaile 0.2.11. For now they’re only available in trunk.

The requirements are basically identical to the ones mentioned in Exaile’s Website, except that you don’t get to apt-get them.

• Python
• GTK+ and libglade for Windows
• PyGTK and its dependencies
• GStreamer, its plugins, and its Python bindings
• Mutagen
• For Python < 2.5, pysqlite
• For Python < 2.5, ElementTree

Right now binary GStreamer Python bindings are only available in the CVS packages, and they’re compiled for Python 2.4 so you’ll need the 2.4 version of everything.

I’m pleased to announce that Exaile gained a new developer last week – Aren Olson. Aren has been contributing here and there for a while now, and as of last week with his new plugin, daap-share.py. What is DAAP you might ask?

Imagine that while at work, you could load up Exaile and connect to your collection running on your machine at home, search through it just like you search through your local collection, drag some tracks over to your playlist and listen away.

You can now, and here’s a little how-to on getting it set up under Ubuntu (Feisty Fawn).

GNOME 2.18 introduced a new way for applications to handle multimedia keys. Previously you have to muck around with X events, while now GNOME does it for you and you can get control of mmkeys by requesting through D-Bus (to GNOME Control Center’s Settings Daemon). All good until you realise that for cross-desktop support you still need the old method anyway—unless, like Rhythmbox and Banshee, your app is GNOME-based.

This article shows how we support both methods in Exaile, and how you can do it, too.