“Intelligence is an ability for slow forgetting”
“Lazy neuron, great intelligence”
“The environment and the Viewer place in that environment is the key player of the prediction game and is the only true and correct point of view for personalised computation of the information”
“The prediction of valuable information for specific User as an average of all Users feelings to the specific information is as same useless as prediction of weather for specific location as average of weather for all known locations”
For a last year I am implementing Artificial Intelligence science achievements into the real world, specifically into the web world. As there are many running projects I would like to add some of my personal observations and solutions in that matter.
My main goal is to reduce the stream of useless informations received by everyone, everyday, everywhere!
I would like to propose the Data Prediction Workflow Model which is able to solve known problems of Streaming Personalised Content to the Viewer at the specific location, environment and point in time via the world web! This model is also able to forecast upcoming Viewer needs.
On the abstract level the workflow pipes all streamed data through predefined plugins like ANN, SVM, Filtering etc. before it actually arrives to the Viewer. The intelligence for pipeline plugins comes from metrics collected on the Viewer UI.
Estimated transmission model of learning is like follows
Intelligent Content Streaming Engine is the platform for Storing, Processing, Distribution and Learning about Viewer to Content interactions within specific environment. The core components are indexers/crawlers for pre-learning, pipelines for intelligent delivery, scenarios for “intelligence” evolving and sensors for detecting the viewer environment.
The engine works in server-client communication model where server stores content as item called “ENTRY” on the list called “FEED” which has own representation on the client UI as the “FEED PAGE”. Communication between server and client instance of “FEED” is bidirectional per Viewer.
storing - is pre-learning about content
processing - is maintenance process for cleaning, growing, rebuilding content
distribution - is intelligent content delivery which combines viewer profile and pre-learned knowledge about content
learning - is evolving process based on viewer and content interactions
The server engine orchestrates basic gears like resource manager, service manager, workflow manager, plugin manager, event manager, cluster manager and manages internal communication.
indexer & crawler - provides indices and metrics about content
sensor - provides informations about viewer specific environment
pipeline - filters content by combining sensors knowledge, content knowledge and viewer profile knowledge (e.g. neural network, svm, bold, filtering, sorting) before pushing to the viewer UI.
scenario - builds knowledge about viewer profile by combination of sensor knowledge, content knowledge and viewer actions (e.g. neural network supervised learning)
Components are packaged as the binary plugins and work separately from engine. Communication is done by RPC and allow for distribution of computation across the cluster. Excluding plugins from engine allows for easy resource management on the OS level for each plugin instance (cpu, ram, networking, hd/).
System Events Management is the key feature of data flow control. It allows for centralised distribution of events during full learning cycle. ICSE implements various low level events and hooks:
new entry - triggered when new entry is added to the feed list
new viewer - triggered when new viewer binds to the feed list
new metric - triggered when new viewer actions comes from the UI to the engine
feed maintainer - triggered periodically; defined by sysadmin or workflow
sensor update - triggered periodically; defined by sysadmin or workflow
push entry - triggered as hook for each viewer UI request
Events are triggered by any of low level engine gears like: resource manager, service manager, plugin manager, workflow manager etc.
store - backend API for application
stream - frontend API for delivery and metrics
metric - backend API for metrics
system - backend API for engine management and monitoring
Service manager takes care of monitoring and scalability of specific services.
flow - defines routes for data through the plugins chain for predefined hooks and events. Allows to limit resource usage for specific plugins.
hooks - binds to stream and store service events
events - bind to global system events
Workflow manager takes care of piping data through defined plugins. It binds deeply to system event manager, stream service and takes care for all "FEED" events on the back-end and front-end.
Heavy prototyping and development of this engine has started on April 2014 and is the running process. Stable beta version will be released shortly as open source...
Db: neo4j adapter (current)
Streaming: websocket, sse, long pooling
- APIs: RESTful
Platforms: Mac OS, Linux, Windows
Backend SDKs: PHP, Ruby (current)
Frontend SDKs: JS (current)
CLI tool (rubygem)
Please contact me if you are interested in meeting the team and contribution to this project in any programming language (go, php, ruby, js, node.js, objective-c, java...). This project is parked on Github and will be hosted at elasticfeed.io
© COPYRIGHT KRZYSZTOF STASIAK 2015. ALL RIGHTS RESERVED