TechNayak(mirror)

Most programmers know databases and its importance. Thanks to the new generation of software as a service and web services, traditional RDBMS’s are sparingly used and the number is bound to deteriorate further as enterprises adopt the Saas platform.

Data has far outgrown the domains of just text. Today we talk of mutlimedia data, urls, semantic data and many more application specific formats. Information on the Web is in JSON, REST , XML , Microformats etc. With this vareity in data formats and representations comes the inherent need for flexibility in storage and querying of such information. Almost all database users know of the conceptual modelling required for the design of any database, the key principle being that more tighter the model, more efficient the database. The integrity of the database is only as good as the integrity of the data. But you cannot talk of data integrity with the kind of formats available today.

Clearly markup data dominates the web . Though databases have developed features to better support , store and validate markup data , the initial design of databases was never to store the wide variety of loosely organized data. Querying of such markup data is fruitless and so is the attempt to index, sort , aggregate this data. To develop a custom database capable of all the above mentioned operations could be a solution, but the given the non standardized nature of this data and its probability of change, you would have a tough time scouring the web to search for changes. Plus these databases will not be semantically inter operable.

Developers are taking notice of a new scheme of storing data, I call it the bucket store. The design is roughly the same as that of a hash table, where data blocks are stored in buckets and hashes are used to index or refer to these buckets. A little improvisation in terms of adding upper layers like domains, groups and so on to complement the schema, table in a database is done to make the data easily classifiable. The advantage with this scheme is heterogeneity in data formats and the absence of constraints.

Several products are offering such services at dirt cheap prices. Take Amazon’s S3 or the recently launched Simpledb or CouchDb which offers a host it yourself version of this storage. Amazon S3 has businesses running on top of it; of the many I can recall Slideshare running on S3. With the advent of more mashups and heterogeneous data being churned out by the web more of such non DBMS related storage options will be employed. Given that this paradigm does implement all the enterprise important features like security, access control , backups, transactions etc and mature modeling methodologies that can rival the ER are proposed , I don’t see any problem in this becoming the most viable and cost effective option for data storage.

There are a lot of schools of thought when it comes to the comet paradigm. There are some who think of comet as a technology in itself that can change the way the web works, while others think its nothing more than another buzzword alongside Ajax and Web2.0. I think, in essence , the latter is more appropriate.

Comet is essentially an design principle which aims at achieving true push technology using http. Comet is not a technology in itself, cometd is an implementation effort to make such a paradigm possible, but comet itself isnt a technology. For real time systems and other applications like stock tickers you don’t need an entire page refresh, its enough if you can get the updated values in your respective places. Ajax seemed to be the answer for sometime with browsers asynchronously requesting for changes and then updating the same in DOM. The problem with Ajax was the polling, since the job was asynchronous, there was a interative pattern of asynchronous requests and replies. Continue reading Comet paradigm implementation…

Note : I blog on my personal space at riteshnayak.com/blog . This is a mirror of the content.
Collaborative apps have been around for quite sometime now, but they have been lurking very close the corporate apps which can be used primarily in a business scenario. A simple example of the same could be the productivity 2.0 apps like Zoho or Google Docs. The only other breed of collaborative app has been games, which is a again a huge draw. Its true that this genre of applications is still finding its foothold on the web and as time progresses you will find killer new applications that will explore new possibilities with colloborative apps.

I had written about Amazon’s Mechanical Turk and how it used the power of collaboration combined with automated project management to get arduous work done from people. Taking and extending on the same paradigm are newer applications that try and achieve some good from these collaborative applications. Its like the Seti project which uses your computational resource when idle, these applications use the power of human intelligence to contribute to a greater cause.

Continue reading Collaborative apps and Collective human intelligence…

Note : I blog on my personal space at riteshnayak.com/blog . This is a mirror of the content.

I read this post by Joshua Porter about the value of social networks and the opaque value problem and it got me thinking. Why and for what do we actually use web based services.

The fact that we don’t understand what value others get from social web apps is part of the paradigm of social software. The key is that each person has their own social lives, their own social circle, and thus their own social values. What is important to their social life will almost certainly be unimportant to us because we have our own to worry about.

Think of it this way. Each person has their own social network. Chances are that social network overlaps very little with yours. If, say, that person wanted recommendations for watching a movie, they might turn to their social network, which is made up of their family, friends, and colleagues. They would ask these people, the people they know and trust, what movies they recommend.

Now, would you turn to the same social network for movie recommendations? Of course not. You trust the people you know…your social network, and so any of the chatter from their social network has no value for you. It’s meaningless chatter. Just like most people’s Tweets on Twitter. In a larger sense, this opaque value problem affects most social software. Twitter, MySpace, Facebook, most social software is built around providing personalized, socially-focused conversation. It’s person-centered and as a result is difficult for anybody but that person to really appreciate: the value becomes opaque in this way. Designing social software is going to be very difficult. The designers won’t be able to put themselves into a position of someone who wants to keep up to date with their social network, which is something that all of these sites are doing.

Social Design  by Joshua Porter  

What the excerpt from the original blog clearly states is that a person using a social software has his/her own reasons to do so. The fact that the others don’t know what value the other person sees in a social network is the entire basis behind social software. I may log into myspace to chat with my peer group, but I dont know what my friend X does in her network, so what do I do ? I snoop around their profiles and walls to get a glimpse of their world coz her value from the network is opaque to me .

The goal of social software is evolve from this paradigm to a more productive one. Efforts are on to make social software enter the productivity space and that too with good success. Based on all of these observations, how do you design for a social concept. How do we actually add value to the users ? Its not easy to please everyone, but at least if the majority is kept happy, you can be sure of winning product.

Note : I blog on my personal space at riteshnayak.com/blog . This is a mirror of the content.

People are always on the lookout for the next software market and a chance to consolidate. But more often than not, a market is entered by a minimum of three players of which one struggles to hold on to the pole position. Take blogging for example, Blogger came in and made things real jazzy but others were soon to follow and now wordpress is giving blogger a serious run for its money. Similarly, in every available space, there are entrants and uniques and the entrants are usually taken over by google

So whats the next big market ? its the $100 laptop. Almost everyone wants to take a shot at something in that project. Even MS got into the act and decided to give out XP for $3. What makes this market lucrative is its sheer number. Its estimated that millions of laptops will be sold and if the project does well, the same will even be extended to third world/developing countries in an effort to provide some exposure and also bridge the cultural and geographic divide. The laptop in itself is a marvel in design. Popularly called XO , this machine is designed with the emerging countries in mind and a lot of work has gone into its usability front.

Yes, its an educational project, not a commercial one, but still once those children grow up and begin to realize the importance of software, they will be clamoring to use better and better. More importantly, its the recognition of a brand that will touch millions of people. Everybody wants a piece of it …. do you ?

If you are a developer and you are interested in knowing more about the XO project and wish to contribute then you can read their wiki here or visit their development portal.

My recent interest in building scalable applications has led me to learn a lot of simple yet effective programming and deployment techniques, to scale and deliver web apps with ease. Its not rocket science to build for scalability but it certainly gets challenging when your app really scales you come across problems about your environment, unknown to you. Take for example our servers, since we host on shared environments our servers flush their memory contents every 24 hours or so. This is done to keep the discarded objects and dangling pointers from filling up the memory. But when did I figure this out, post deployment - what followed next was an analysis of all the problems that we were to face because of this new found wisdom. Truly, most of our so called cool functionality came crashing down like a hailstorm ( truth behind why Instant Messaging never got off the ground on Samparkh ). We did try to tailor the code and make it work, and it did too, but a problem like this requires a visit back to the drawing board. The next time we know better than to code with assumptions ; most of us aren’t lucky enough to know all about our deployment environments, so we make do with assumptions.

That was a lesson, no harm done, no money lost. But there are some applications that have to be delivered to be robust , scalable and always available. I found this article which highlights some rules for high performance website. I suggest you visit this site and read about these rules, its worth it .

• Rule 1 - Make Fewer HTTP Requests
• Rule 2 - Use a Content Delivery Network
• Rule 3 - Add an Expires Header
• Rule 4 - Gzip Components
• Rule 5 - Put CSS at the Top
• Rule 6 - Move Scripts to the Bottom
• Rule 7 - Avoid CSS Expressions
• Rule 8 - Make JavaScript and CSS External
• Rule 9 - Reduce DNS Lookups
• Rule 10 - Minify JavaScript
• Rule 11 - Avoid Redirects
• Rule 12 - Remove Duplicate Scripts
• Rule 13 - Turn Off ETags
• Rule 14 - Make AJAX Cacheable and Small

Note : I blog on my personal space at riteshnayak.com/blog . This is a mirror of the content.

All you javascript programmers, are you one of those people who are bitten by the RIA bug and write endless lines of JS code and include 200Kb header or bootstrap files. Then you suffer from bloat, a condition in which a content 3kb in size brings with it 300kb of javascript for presentation purposes. Bloat is exhibited by Gmail or any of the latest google applications , which while opening sometimes makes your browser really sluggish and non responsive.

Its not so bad in the real world. There are techniques being thought of to reduce the memory footprint of these mamoth js bootstraps. First step would definitely be to use the right library , if you use dojo make sure you use the right flavor of dojo like dojo for ajax or dojo for UI etc. There is no point in having functions bloating if they arent used at all. Next step would be use some brains and filter out unwanted functions in your bootstrap files. Its not that hard I guess, get your self Venkman or the recent firebug and you can be on and reducing in no time.

Now for the real deal:

1. use Dojo Shrink safe to reduce, compress and pack all your js files together. Most of the times your file reduces by almost a third using this. This utility doesnt obfuscate so you can call the same methods and not worry about changing function names.

2. Use Dean Edwards packer to further reduce the size by removing linebreaks and other unwanted characters. This usually reduces the file by another 20%

But make sure you always have a backup of the files. As most of these things are unreversible. These are just performance optimizers, they dont help your programming.

In the worst case, if you still a Kb or two short of your SLA then go ahead and obfuscate your code. Packer does a little bit but there are some really neat ones out there that can do a wonderful job. whats the advantage you ask ? a function OnWindowFocusAndDoubleClick() will get replaced by say g() , now thats improvement.

I found this utility which does reduction, try it out : jsjuicer

 I have moved my blog to my personal space but continue to mirror content here because there are many who still read this blog. I see a lot of feeds pointing to this blog rather than my new space. Please update your bookmarks and RSS subscriptions to my new space.

My blog: http://riteshnayak.com/blog

RSS Feed : http://riteshnayak.com/blog/feed

Who am I? : http://riteshnayak.com

If you are not the RSS type and also don’t find the  time to read my blogs, get it delivered by mail. Sign up by clicking on the link below and all my updates will be sent you by mail

Get my blogs by mail here : http://www.feedblitz.com/f/?Sub=192584

I just read an article from the web2.0 expo where Jeff Bezos of Amazon gave a talk on how Amazons S3 service , which essentially allows you to use Amazon’s servers as your data store, crossed 2 Billion stored objects. For a meager sum of money you get the scalability and the reliability of Amazons servers, basically the same architecture Amazons own services are based on. Thats where he announced “Infrastructure as a service has arrived“

I did some research on S3 sometime back when I intended to use it for some project, and found it really compelling to use. There were lot of problems reported by users and some outages also which hampered lot of sites who depended completely on S3. There is also the problem that your traditional taxonomic structure that you were working on will now be replaced by a Hashtable like datastore that you essentially query. I found it a great way to store shared media, photos and other non text based information and rightly so. S3 fits in as a replacement to a database in the more traditional sense, but it has its own limitations, but its power nevertheless.
So what is this IAAS( see title) , according to me its the death of the network engineer.Gone are those days when you would rent our servers and rope in DBA’s and Network engg who would manage the hardware and availability part of your application. You want unlimited computational scalability? try mediaTemple’s Grid server, want unlimited datastore? try S3, want scalable content delivery? try Akamai Edge. Infrastructure management is at an all time low in the 2.0 era, especially since technology is more easily available , manageable and scalable. Infrastructure is also turning into another SOA where pre built components define how you develop and deploy your applications and you don’t have to worry about downtimes, reliability, scalability and other metrics that are considered important for any web service.
Whenever such disparate elements contribute to a unified cause the next step is  consolidation. But will it ever happen ? these service providers bank on the fact that you need these systems to run your application, otherwise you wouldn’t need a database or a blade server. Have services like S3 given companies like oracle a run for their money? Will there be a similar offering from the other majors?
Picture Oracle providing database hosting on their enterprise level infrastructure for a minimal cost. Updates and upgrades done automatically, maintenance not a pain anymore. Its a space worth debating on and worth watching out for.

Making web apps secure has been an effort ever since the advent of web
applications. The strange mix of technologies interwoven with the complexities of each one of them hasn’t encouraged the setting up of standards for these. Scripting hacks, Cookie hacks, cross site scripting, malicious users and other common problems plague web applications. If all these aren’t enough, you have denial of service and other scalability problems. There are just a handful of best practices and tips that you could follow to make your web applications more secure. I shall list some elements down from my experiences.
 
   1. Validate all data that goes that goes to the server
        Make sure you validate all the data that goes through the browser, it could be forms you are submitting with critical data or some event like clicking a link. Make sure your dataset that the server receives is what it is intended to receive. In case of bot threats, use captcha’s and simple questions that require human intelligence and natural language processing. By validating data, you are just saving yourself that many less attacks on your server.

  2.  Use your JS wisely
         I have seen a lot of apps currently using a wealth of js libraries, and since your js needs url to access, the XMLHttpRequest object with its url is wide open for the entire world to see. You dont even need to look at the source , with a sniffer like firebug, you can see all the async requests made, with the url and the response. Scripting attacks are the most prevalent of the security attacks and people can do wonders with malicious scripts. The trick here would be to confuse the hackers. Obfuscate the scripts wherever you can, there are good JS obfuscating libraries available. Be advised , obfuscation is usually irreversible , so use it wisely. Use POST instead of GET for async requests and make sure its the right person requesting the data. You could probably generate a sessionid of sorts on the server end and pass it to the browsers as a key to use for any further requests. And please, don’t rely on cookies, they can only go this far.

  3. Follow the MVC methodology
      it usually helps in abstracting the finer points of the application. Separating out the data, logic and accessibility will usually give you more control on what goes where and a lot of server side validation.

  4. SQL Injection
   I understood the complexity behind these attacks only recently. A seemingly simple line of DB code could prove disastrous for applications. Always use good wrappers around your db code. Many good SQL Helper libraries help you against such attacks and the overhead is adding another server side library, which is not so big considering it can save your application.

 There is also the new generation of security threats thanks to Ajax and is open ways. But there are safer alternatives being developed like JSONRequest which abstracts the JSON based calls  and provides a safe way of making sync calls.

  I shall write more about web based security when I get time.