http://russellbeattie.disqus.com/... because you can't shut up.enWed, 09 Jul 2008 01:04:08 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-842101Here is my rant, <a href="http://siliconvalleymusings.com/164/twitter-scaling-issues" rel="nofollow">http://siliconvalleymusings.com/164/twitter-sca...</a>, on how Twitter could be designed to scale, identi.ca could use the same approach.SteveWilhelmWed, 09 Jul 2008 01:04:08 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-831166This really isn't that hard to scale large - as you point out with your mention about queues. If identi.ca implements an async message queue, they can scale quite large. The last project I worked out handled north of 1 billion msgs/month, so it can scale up quite a bit with not much effort.Dave HodsonMon, 07 Jul 2008 19:02:08 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-828911I know that Friendfeed appears to be more about aggregation than microblogging but I'd be interested to know whether the architecture for Friendfeed suffers the same basic design flaw described in this post. Good to see Evan jumping in and trying to overcome all the obstacles by the way.andyrobertsMon, 07 Jul 2008 14:25:55 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-826018Queues do not replace the DB though, it sounds like that here. You still need to store things somewhere, indexed, and databases are useful for that, although relational ones are probably not a good fit for this kind of data, and "inifinite" scaling. There you would probably need something easily like bigtable, hadoop, hypertable or whatever. I guess you could also partition a regular DB, but I am not convinced that is as easy, one you got it all going.sandosMon, 07 Jul 2008 07:11:06 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-825386You're of course completely correct. I can't, in fact, see why any page on either Identi.ca nor Twitter needs to be dynamically updated upon view. They can all be 100% static .html files, re-generated by a the batch job you describe, every now and then. This static file regeneration could be made very effective and would be extremely easy to scale.asbjornuMon, 07 Jul 2008 02:35:22 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-816056One last thing, Russell: you may not be familiar with the XXX/FIXME commenting tradition. "XXX" means "I know this is bad, but it's good enough for now, so I'll come back and fix it later." And "FIXME" means "don't let this out the door". I don't think there are any FIXME comments there, but you can grep for XXX to find places in the code where I think optimizations are in order.<br><br>One thing in particular: I had problems with making joins work in DB_DataObject, so there are places where I have nested loops instead of much-more-efficient joins.evanpFri, 04 Jul 2008 19:28:54 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-815234i don't get your comment about services like twitter needing a messaging architecture.<br><br>twitter uses this architecture right now and it still doesn't work. <br><br>tweets are not published automatically for instance, they use starling, a daemon whose task is to take all tweets and process them. The subscribed users then get the updates. exactly how you described it. <br><br>but this is flawed. why? because when you have thousands of followers, the message has to be copied thousands times. imagine what happens when another user who has thousands of followers replies back. i'm sure even smtp would also choke on that -- it's not designed for users wanting to message thousands of addresses. <br><br>that's why i think identi.ca is not better or worse than twitter. but here's one thing: users can install it on their own servers helping thus the load to be distributed. which is a great architecture design. <br><br>note: i don't necessarily have the solution though.heriFri, 04 Jul 2008 15:51:09 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-815101Russ, you're right on the money. We had the very same discussion yesterday on NewGang Live. First one to actually do this as a true message bus wins.jschumanFri, 04 Jul 2008 15:12:57 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-814386What is the problem again with a mid table that has the user's id and the id of everyone they are subscribed to as the limiting query for the user.all_subscribed_messages?beingbradFri, 04 Jul 2008 12:37:31 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-813401Patches welcome, Keith. (I'm serious; that's why it's open source.)downlodeFri, 04 Jul 2008 09:08:04 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-812739Sure, let's figure out a way to share it. Once I'm over the hump on this queueing thing tonight (you can pull the code from darcs right now, by the way), I'm going to bed for about 14 hours, then I'm going to respond to all the great code contributions I've gotten over the last day and a half.evanpFri, 04 Jul 2008 04:11:26 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-812727Please, let me know if you need any help. As you can already see, there are some places for storing multimedia files (urls, mostly) in the DB already. It'd be great to take account of licenses, since one key part of Laconica is to support sharing free content.evanpFri, 04 Jul 2008 04:08:01 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-812722Does it need to be in a repository right away or do diffs count? I don't mind doing an export of the code into Google Code or something, but it'd be pretty silly based on the number of lines I've tweaked (probably all of 10)...<br><br>-RussrussellbeattieFri, 04 Jul 2008 04:05:26 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-812717Evan!<br><br>That's great to hear about the queuing! <br><br>Of course, you can have access to my changes (as per the GNU Affero license. ;-) ), but honestly, I haven't done anything except tweak a couple lines of PHP here and there as I work the through the code and figure it out. Working out where to splice in queuing would be way beyond my understanding of your code at the moment, though I'm happy to help as I get to know how things work. <br><br>-RussrussellbeattieFri, 04 Jul 2008 04:03:58 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-812711Also, where's your source?evanpFri, 04 Jul 2008 04:00:39 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-812708Hey, Russ! I'm busily working on queueing right now, as we speak. Seems like we're wasting effort on competing codebases -- any chance you'd like to re-integrate and coordinate our efforts?evanpFri, 04 Jul 2008 03:58:31 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-812554Quick question? do you know who on identi.ca wanted a version for their classroom? We are building a microblogging service at <a href="http://edmodo.com" rel="nofollow">http://edmodo.com</a> for teachers to use in their classroom.zemoteFri, 04 Jul 2008 02:57:03 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-812240I think part of the Laconica story, though, is getting the software installed on commodity PHP web hosts for federation. That kinda quashes a beanstalkd installation for most of that audience. Doesn't preclude it as a later bolt-on option, thoughlmorchardFri, 04 Jul 2008 00:51:00 -0000http://www.russellbeattie.com/blog/let-the-microblogs-bloom#comment-809943And the really silly thing is that there are perfectly good FOSS message queues out there like beanstalkd that they could use.Keith GaughanThu, 03 Jul 2008 17:23:05 -0000