TechNayak(mirror)

Is it true? Is Indian IT industry surviving because of lack of Semantic Interoperability?

Incorrect Semantic Interoperability describes mismatch in the format and representation of data belonging to two parallel applications which prohibit them from interacting with each other or prohibits possible migration. Take the example of ERPs like SAP and Oracle Apps, they essentially perform the same computations and solve the same problems, but do they interoperate? No. The data representations for both these applications are bespoke, which makes it unique to one product or a line of products. It is reasonable considering the fact that market domination is obtained by developing custom formats and providing custom decoders, but there is a bigger concern. For large players, data is collated and there is usually a Business Intelligence solution in place and with different products being deployed at different centres, the additional overhead of conversion of the data is imminent. Also two separate applications cannot talk to each other though they are probably linked up sequentially, you have to bring in additional middleware for format conversion and make a common bus.

Efforts like RSS, SOAP and others have been successful only to a certain extent that the format is correct but semantics are still loose. It has to come together sometime, if not now then later, when the data is huge. And the answer is yes, the Indian IT industry survives because of such semantic interoperability. Most of the workforce is maintaining these enterprise bridges (that holds together these different applications) if not building them. A majority of the work involved in the service sector has to do with writing compatibility plugins or writing migration scripts and patch ups. If it was for SI , we wouldn’t have jobs.

Gone are the days of the digital divide, there is a new kind of divide amongst many computer professionals now. Its the generation gap. Its hard to comprehend this statement, but anybody, whose is exposed to at least 5 years of industry dynamics, will know exactly what I am talking about. Call it Moore’s law affecting software or just plain old generation gap, there is a clear demarcation between people who appreciate new concepts and those who prefer things the 90s way.

There are a set of people that like the innovation happening on the web front and are adopting 2.0 technologies like there is no tomorrow. Everything from office automation to project management is now managed online on productivity service providers. Concepts like wiki, blogs, forums etc are fast appearing as mainstream applications in organizations. Surely as technology evolves and takes new shape, we will see a dramatic shift in adoption of these new tools .

In contrast , there are the other people who have been around for a long time and have seen a lot of productivity applications. To these people, technology is nothing more than a fast changing fad and prefer to stick to their old time favorites. Take people who have seen the main frame era, such folk just don’t appreciate concepts like distributed computing, virtual servers etc. Quotes like ” our mainframes never needed mirroring”, are common. People who still live reminiscing innovation of their times like spreadsheets and ERP’s.

It may be hard to believe but these form the majority of the so called power users of organizations and these legacy softwares( pun intended) , are maintained and supported just for their usage. Its distrubing to know that enterprise software lags open source software by at least 3 years , in terms of innovation. This lag can clearly be accounted to the legacy lovers who insist on using their accustomed softwares. Where does product development go in such a case. Office 2007 is seeing very slow adoption due to a change in the usability. Will this set of users be responsible for the sluggishness of product development? who will convince these users to adopt newer software? more importantly how? What will these users demand 20 years from now?

Its a strange question, but yes its an emerging market.

I wrote about personalization some time back and about how we should actually be approaching this problem. Google has got their act into place and are making your own light weight personalization meter, but its for ads

Google is going to put a cookie in your browser that will record information everytime you read an ad served by Google. Continue reading Personalization is one cookie away…

There was an interesting concept put forward to me by my professor - Open World Computing. A software that is not bound by any restrictions or constraints. A software that learns and adapts to its environment. Think of a person who is taken from a metro and put in a village. Does the person fail and give up like a computer program ? No. the adaptability of living beings is something so hard to understand that it can take probably another 1000 years to just simulate a living being, let alone learn its qualities.

Think of the same thing in software. A software that is programmed based on generic constraints and the software dynamically learns from its environment through input devices like sensors and then adapts to the changed environment. The classical shortest path problem can be taken as an example. If a crawler running through the shortest path is lifted from its path and put somewhere else, will it be ale to comprehend the change and then quickly adapt or will it be lost. What if the graph changes and produces a lot of cycles or what if the problem statement changes during the course of the program.

I know that a very few people are relating to what they are reading, but the belief that a machine can learn and adapt is what scientists are trying to prove everyday. Machine learning is taboo after movies like iRobot and Terminator, but trust me, we are far far away from something like that. If we can solve a subset of problems of adaptability through an expert system, that will be an achievement in itself.

But how do you go about designing software with such requirements. I would go one step ahead and call it no requirements or changing requirements with no defined thresholds. No current methodologies like OO or aspects can cater to such a requirement. Probably a new scheme of designing learning software has to be developed. Taking tips from AI and the Turing thesis, a perfect turing machine is what is required. Its a hard call but we will get there one day.

Note : I blog on my personal space at riteshnayak.com/blog . This is a mirror of the content.

The recent opening up of the facebook platform has created a rage in the industry. Facebook themselves , after opening up to public signups, have had an increase of 80% in their monthly uniques. The facebook platform is just another icing in the cake.

There are innumerable number of apps that are being created on the facebook platform. Existing applications like wordpress are taking heed and porting their apps into the platform. Some of the initial apps have been taken over and VC’s are announcing seed fund for apps built on the facebook platform. Its like everyone wants a part of the 26 Million uniques visiting the site and want to cash in on the phenomenon. So what did facebook do right ?

Continue reading Community as a Service - implications of the facebook platform…

Note : I blog on my personal space at riteshnayak.com/blog . This is a mirror of the content.

I read this post by Joshua Porter about the value of social networks and the opaque value problem and it got me thinking. Why and for what do we actually use web based services.

The fact that we don’t understand what value others get from social web apps is part of the paradigm of social software. The key is that each person has their own social lives, their own social circle, and thus their own social values. What is important to their social life will almost certainly be unimportant to us because we have our own to worry about.

Think of it this way. Each person has their own social network. Chances are that social network overlaps very little with yours. If, say, that person wanted recommendations for watching a movie, they might turn to their social network, which is made up of their family, friends, and colleagues. They would ask these people, the people they know and trust, what movies they recommend.

Now, would you turn to the same social network for movie recommendations? Of course not. You trust the people you know…your social network, and so any of the chatter from their social network has no value for you. It’s meaningless chatter. Just like most people’s Tweets on Twitter. In a larger sense, this opaque value problem affects most social software. Twitter, MySpace, Facebook, most social software is built around providing personalized, socially-focused conversation. It’s person-centered and as a result is difficult for anybody but that person to really appreciate: the value becomes opaque in this way. Designing social software is going to be very difficult. The designers won’t be able to put themselves into a position of someone who wants to keep up to date with their social network, which is something that all of these sites are doing.

Social Design  by Joshua Porter  

What the excerpt from the original blog clearly states is that a person using a social software has his/her own reasons to do so. The fact that the others don’t know what value the other person sees in a social network is the entire basis behind social software. I may log into myspace to chat with my peer group, but I dont know what my friend X does in her network, so what do I do ? I snoop around their profiles and walls to get a glimpse of their world coz her value from the network is opaque to me .

The goal of social software is evolve from this paradigm to a more productive one. Efforts are on to make social software enter the productivity space and that too with good success. Based on all of these observations, how do you design for a social concept. How do we actually add value to the users ? Its not easy to please everyone, but at least if the majority is kept happy, you can be sure of winning product.

Note : I blog on my personal space at riteshnayak.com/blog . This is a mirror of the content.

I suggest : Use the internet to make your international calls with VoIP calling service. You can download free VoIP software or get a VoIP phone to get started. You can even become a VoIP reseller selling VoIP service.

How do you define evolution on the internet ? When experts predict how the internet will evolve and what the future trends will be , what are the factors that are taken into account. 10 years back, no one could have predicted Social Networks and blogs , but look where we stand today. Evolution can mean a lot of things we could evolve with respect to …..
Technology. The core technologies can change. From HTML to web services to Flash , the technologies that make the foundation of the internet can change. The methods in which we access the internet can change , course, its the technological aspect. We have wireless internet, we have BPL - bandwidth through power lines. The means in which we access the internet can change. We access internet from PCs, laptops, mobile phones, handhelds, kiosks. Widgets were probably never even though of 10 years back.

How about usability ? the way people use the internet has changed dramatically. Tons of data being uploaded everyday. Almost all of information of a person can be found on the internet. Thats why any criminal is always looked up in myspace or other social networks. Also how people interact with internet applications is also a factor to consider. Today we have IM, telephony, music players, media libraries, wikis, photo sharing services, blogs, and many other ways to interact with the internet. Who knows? probably in three years time you can transfer money from one account to another by just sending an IM through your favorite client.

Its not just we who perceive the internet. The way the internet perceives users has also changed. Its not just X searching on google, its X who lives here, likes this, hates that, has searched for Y before and is more probable to buy Z. The future could hold machines that can identify with you, relate to you and even deliver specially tailored content to you. The web has also become extremely customizable with every site offering you the concept of skins and templates.
What bout user behavior and usage? ten years back double clicking on the browser wasn’t even a concept. Autofilling textboxes and combo boxes were a fantasy. and Drag Drop, well it wasn’t available for some the best desktop apps, forget the web. Today people use them like they have been around for ages.
what about user evolution ? I have gone on from becoming a mail checker -> sleeze surfer -> information hunter -> programmer -> webmaster -> blogger and I have missed out probably tens of roles. How do we account for user evolution and experience.
Its pretty fuzzy for me. How do you define evolution on the internet ?

Note : I blog on my personal space at riteshnayak.com/blog . This is a mirror of the content.

Inspiration for this post : How much textual information we consume everyday

Text makes the web. Those characters that are 1s and 0s in their stripped down version form the basis of what we call information. Right from websites to search, feeds to news, wikis to chat they are all text. Its unquestionable that people consume a lot of text on an everyday basis and a simple look at your bandwidth bills will show you just how much. Why am I ranting on about these things ? Well, it so happens that the textual information that we consume actually help in realizing a very big dream, the dream of personalization.

Personalization, the word is one of the hardest to define and yet very simple to perceive. Its the need of the hour, no matter where you go, there are people craving personalization. As developers, our view of personalization is very limited. We tend to be more involved in the visual aspect of things like the color, the theme or at best the layout. These are akin to having balloons and glow bugs on your workplace , nothing more than visual appeal. What we must strive is the personalization of information and not its presentation.

Thats really hard : no it isnt. Take a look at my RSS reader and you will be able to judge what kind of information I am interested in. Why restrict it to only blogs. Take a look at my mails, my socionet profile, my friends data , my search results. All these have some details of the data that I consume everyday.

Why do we need that : Its a time saver, in terms of finding the information that I need. My search results can be more inclined towards the information that I am interested in. My social network can show me more like minded people rather than an arbitrary selection. Why even the ads can be targeted more towards things that I buy. I find the biggest ROI in terms of advertising. Let me explain with an example: Supposing i bought an Ipod , I download the itunes software, read abotu the instruction manual, next search for some popular tricks, themes, games etc. Now supposing I were to go to ebay and click a button “Show me things to buy” , based on my previous surfing trends its not hard to predict that I would want but accessories and other iPod related stuff. Now isn’t that what personalization should be about.

How do we do it: Its not practical to expect a Google or a Yahoo to do all the personalization, of course Google is taking serious strides towards personalization in its truest sense. The system has to be decentralized, it should be ip based, even if multiple users use a system, there is more likely a chance that their geographical location has a pivotal role to play in their surfing habits( take a hostel for example, more often that not you find people with similar necessities on the web) I think an OpenId like implementation of a personalization filter would definitely help a lot. You have distributed servers that keep relaying information about a certain persons information consumption habits and then when you ascertain certain patterns, you writeback to the server the pattern. Upon querying or any other action, you get a collated version of your pattern and then make decisions based on your content vs the required content. Aint it a cinch ?

Truly 2.0 has been about You but 3.0 will be about me….. everything me.

Google unveiled their universal search page for their search results, basically a page where you get more than just web pages, you get blogs, pics and videos even. But this is my question - is the concept of universal search even possible ?

Penetration of user generated data into spaces like technology and entertainment have been exponential. Every person who wants information on code and solutions looks up forums and blogs but there are certain niches which are still firmly controlled by the creators like astro physics and rocket science ( pun) . Now when we have such a divide how can we make sure that universal search is actually delivering to its full potential. The primary criteria I guess is to put the search query in its proper classified subset. By this I mean, if I search for a topic that has more material in a blog than a wiki or a full blown site, then how will the algorithm know its relevance. Ranking clearly fails in these conditions and its pretty evident.

The second biggest problem is of course, the real estate. What I may be looking for may not turn up in the first search result and with most of my observation people don’t like shifting pages to more results, they decide to change the query instead. In that minimal screen how will you fit in the right contents based on the rank, relevance, semantics and many other jargons that even I, haven’t heard of. One thing clear though is you need to mine.

Mine data from the users, their previous search results, probably from the one place where they have disclosed something about their life, their interests, hobbies etc - Social Networks. Even the real estate management should change , there should be a new scheme of viewing results for a query, probably a tag based scheme that can subdivide the query into many groups and subsets of relevance and then refine it on top of the tags. It requires a change, in peoples usage, which is a dreaded de-motivator , but tags started the same way and its doing pretty alright in sites like flickr and del.icio.us.

The concept and implementation of web search is still unexplored and untapped. There are a wealth of things that can be done and I have mentioned a couple above. Semantic web could solve some major problems, but its still a long way away and more time means more space for innovation.

At the Big Thinkers conference by Andrew Tomkins of Yahoo, the word interestingness caught my attention. Though the idea of judging the interestingness of an object like a photo, document, video seems far fetched, I nevertheless was fascinated by the possibilities of this new metric. I found this patent document filed by Yahoo for their interestingness concept in Flickr.

claim 1: An apparatus for determining an interestingness rank for at least one media object, comprising: logic for accepting at least one metadatum concerning the at least one media object from at least one user; and logic for ranking the at least one media object based at least in part on the quantity of user-entered metadata concerning the at least one media object.

The apparatus of claim 1, wherein the user-entered metadata comprises a member of the group consisting of: tags, comments and annotations.

Well there are new web metrics being rolled out almost everyday and its only time, at least some of them, become mainstream. One of the strong contenders I feel is the interestingness metric. Based on the number of bookmarks, clicks , click throughs, views, tags, annotations, visits, visit length and many many more factors interestingness can be defined, although this may not be Flickr’s definition. Any piece of information like a music piece , a podcast, a blogpost, a video clip, a picture etc can be associated with interestingness - It simply gives the most popular item amongst a list of inventory. These items can also possess geographical information, time line and other information that can be used to extract essential patterns from these items.

The usage, could be a phenomenon on its own. Define a properly ranked algorithm and you could have your top selling items on your e commerce site. You could target geographically distributed audience with content of their choice and also give interesting timeline applications that let you see trends over time. Say for example you run a shopping portal, you could give a time line graph highlighting your most sold items over a period of time. This could also prove a big step in the analytics area where ROI and conversion from visitors to buyers play a significant role in everyday business and concepts like these could really boost your sales. Its still unsolved from a very broad sense, meaning there isn’t a standard set of practices to encompass such a wide variety of data and there isnt such capable systems that could mine patterns out of these collations. But its definitely interesting