The Internet of Things: Big Data Making Smart Farms in Kenya, Africa

Posted On // Leave a Comment



Enter the Internet of Things, a disruptive technology predicted to change almost every aspect of our lives, from health, education, security, entertainment, business, governance to agriculture in the near future. It could help change lives by raising food production in sub-Sahara region where 95% of arable land relies on rainfall-fed agriculture, and no one could doubt the fact that climate shocks over the last couple of years have worsened yields.



The internet, remote sensing technologies and cloud-based data sharing platforms are only beginning to change the way we see and do agriculture.Call it precision agriculture in IBM's own words; It uses big data and the Internet of Things to provide insights into current and predicted water and soil moisture levels to farmers and water service providers, who then act as necessary to respond to the situation.

The information is delivered to end users through apps installed on smartphones and tablets. The EZ Farm is an IBM project, which is on trial in a number of farms in Nairobi, Kenya involves placement of sensors that collect and stream data to IBM’s cloud-based data centers and updates the information every minute. Farms are equipped with water tank sensors, soil moisture sensors and infrared light sensors (to monitor health of plants). The result is high yields and low farming costs.

Empowering the farmer

While many governments have already implemented means of weather predictions based on the recent technologies such as the satellite and remote sensing and are able to put into place crucial response measures mostly in form of advisories and directions to farmers, the next set of challenge still stares at us: how to empower the farmers to react appropriately based on first hand and immediate information.

It appears true revolution will come when such first hand information directly reaches the farmers and helps them implement immediate and necessary measures on their farms other than just waiting for advisories only from governments and agricultural organizations as has always been the case. This is made possible by the Internet of Things.

Precision Agriculture, Courtesy of IBM
Image: Precision Agriculture, Courtesy of IBM

The EZ Farm is an IBM project was particularly designed to help a group of tech-savvy farmers known as "telephone farmers" who dwell in urban areas and only travel in rural areas during the weekends to watch for their farms. It helps them to better manage water resources needed to irrigate farms as well as to monitor crop health. The application has the potential to bring revolution to small scale farmers according to IBM Research - Africa V-P Dr Kamal Bhattacharya.

“Kenya has 15 million small-scale farmers who contribute hugely to the country’s GDP but we are still investing on food security,” said Bhattacharya.

EZ Farm project is also a leading example of how the Internet of Things can help manage water, a resource sometimes so scarce in some parts of the world. Dr. Kala Fleming, Ph.D. Civil and Environmental Engineering at the University of Wisconsin who leads the leading the EZ Farm project says he wanted to spend time thinking about how water and natural resources can be harnessed to build communities and describes the success of the project. By putting water into best usage at the farm, availing it when needed, lowering water consumption and wastage, the project helps farmers get the best out of their projects.

"The idea that we can get people to see how they can improve yields through water management, while also having a younger generation get excited about being involved in the water industry are all things that point to success for me," said Fleming.

More data from diverse sources

It is expected that future farmers will benefit from diverse data collected from as many platforms as possible - from satellite, farm sensors, marketing, internet and cloud, and delivered on a single platform such as a mobile phone application.

Monitoring and taking advantage of data collected by satellite alone can help understand soil moisture content, precipitation, and vegetation health, and help farmers make necessary and timely decisions. The EZ Farm is also hoping to hook up data from the Kenya meteorological department to include information about rainfall patterns and predictions, which will be provided to farmers through the mobile applications. In future, farmers using this IBM technology can expect the sensors to transmit more data about their farms, for instance real time operating status of farm equipment, such as pump failure. This will help save costs further.

The fact that such data is dispersed to clouds such as the IBM Cloud means organizations can also use the Internet of Things technologies to provide advisories and farming education to farmers through mobile platforms according to Flemming.

"Creating a digital network of small-scale farms and water users also provides opportunities for other organizations looking to launch value-added services to generate revenue and increase productivity," Fleming told BBC.

Some projects that deal in farming insurances are already taking advantage of Internet of Things to deliver better services. For instance, using historical satellite data, micro-agricultural insurance scheme ‘SUM Africa’ (Scaling Up Micro-insurance in Africa) based in Mali and Uganda is able to predict yields and compensate farmers who are policy holders if predicted yields fall below a certain level. The compensation is done early enough that farmers can take necessary measures to shield themselves from expected calamities. If those calamities happened, they avoid lengthy calamity assessment procedures that preceded compensation.

SUM Africa; mage courtesy of World Bank
Image: A tomato farmer in Ethiopia sorting out products, Coutesy of World Bank

An Index-Based Livestock Insurance program in Northern Kenya uses satellite images to predict "greenness" of pasture and farmers who take insurance policies can actually get financial compensation if the satellite shows there is little pasture left for the cattle.This money can be used to protect livestock against drought by taking necessary initiatives.

Another case in point is the use of big data and the Internet of Things to empower farmers by giving them immediate and timely access to marketing data, which allows them to take necessary decisions and harness more benefits from their yields. An example is the M-Farm which leverages on internet and text messaging to help Kenyan farmers access market prices for produce, aggregate orders of farm supplies and thus lower the purchase costs, and sell their produce in bulk thereby reducing related marketing costs. Farmers could still link up with mobile-based financial services.

Smartphone applications such as the iCow are also being used to help farmers gain access to educative content such as feeding practices and disease control; record keeping tools; and connecting to buyers. FarmDrive is a simple record-keeping platform that also helps farmers to access credit services while the Short Messaging Service-based WeFarm application helps farmers by troubleshooting their farm problems by sharing information. The MbeguChoice application helps farmers to access better, drought tolerant and variety of seeds from various suppliers.

However, the real modern challenge is the actual number of people owning smartphones especially in the rural areas, which is definitely expected to change in the near future. If the technology benefits the first starter tech-savvy farmers, it might spread.

Cheap sensors, inexpensive remote sensing imaging from satellites, cloud computing, and information sharing via mobile device applications will drive the next phase of data-driven agriculture called Ag 3.0 by Lance Donny, founder of an agricultural technology start-up, OnFarm Systems.

“It’s a totally different world than walking out on the farmland, kicking the dirt and making a decision based on intuition,” said Donny during his presentation in a workshop hosted in San Jose, California by the National Science Foundation and the National Consortium for Data Science.

Courtesy Of David Kariuki
[Read more]

Smart Cities Of The Future - Making Cities Smarter Using IoT Sensors

Posted On // Leave a Comment
Making Cities Smarter

MIT researchers are creating tools that synthesize and collect data so that urban planners can vastly improve the quality of urban life.


Imagine your city as it might be in the not-so-distant future.



Transportation in this city is various, pleasant, and low-impact. There are safe and efficient bike lanes, and anyone can order a cheap ride from an autonomous, minimal-emissions vehicle. Because fewer people drive, and almost no one idles in traffic, air quality is high. There are plenty of parks and open spaces because cars are less prevalent. Life in your city is happy, healthy, and sustainable.

Your City Is, Above All, A Smart City.

The smart city, like the smart home, is built on and around the “Internet of things,” in which networked products gather, store, and share user data while communicating with one another in order to create improved and highly-efficient living environments. In a smart city, the Internet of things expands outward from the home into a plethora of automated and interconnected urban devices. The communication between and among these devices allows for vast amounts of municipal data to be gathered and eventually analyzed. A smart city leverages its collection of massive data to learn about its residents, showcasing the ways in which smart cities are beginning to transcend the Internet of things, by gathering massive data sets that are gradually helping researchers understand vast and complex networks.

However differently smart cities may be defined or described, underlying them all is an array of interconnected social networks and systems, an understanding of which allows for data-driven urban planning that stands to vastly improve the quality of urban life. Sarah Williams, an Institute for Data, Systems, and Society (IDSS) affiliate and assistant professor of urban design who directs MIT’s Civic Data Design Lab — an urban studies center that uses both data visualization and data collection to identify and understand various urban phenomena — is an example of an urban planner using this data to communicate the complexities of urban life in order to drive decisions. “When data is made comprehensible to a large number of people,” Williams remarks, “it is well-positioned to drive social change. Creating tools that synthesize and collect data transforms how we see the world, at one time showing us the effects of policies while also providing essential information to develop new urban strategies.”

William’s research shows the kind of impact IDSS researchers are having by developing and communicating an understanding of vast and complex urban social networks. At the same time, other IDSS researchers are helping to develop smart technologies that will power future cities, such as autonomous vehicles and smart energy meters, using a systems approach to build effective solutions for the improvement of urban life and the solution of societal problems.

Transportation Systems in Smart Cities



Transportation is one of the greatest of those problems, and one of the most essential areas for innovation within the smart city — particularly the promise of autonomous vehicles. Emilio Frazzoli, an IDSS faculty affiliate based in the Department of Aeronautics and Astronautics, has made significant inroads in the area of autonomous vehicle innovation. Frazzoli joined project leader and senior paper author Daniela Rus, the Andrew and Erna Viterbi Professor in Electrical Engineering and Computer Science — as well as other colleagues — in testing an autonomous vehicle pilot scheme last fall in Singapore, where the Singapore-MIT Alliance for Research and Technology (SMART) is based.

Over six days, autonomous golf carts were made available to visitors in a large, public garden in Singapore, where passengers could summon them through an online booking station and book rides to and from predetermined points. The small carts, a minimalist version of an autonomous vehicle with a maximum speed of 15 miles per hour, adroitly navigated paths in the garden, making sure to avoid pedestrians and cyclists.

Frazzoli is now working to create street-ready autonomous vehicle technology that could transform urban travel in the near future. “If deployed more broadly,” Frazzoli remarks, “autonomous cars have the potential to change how we think of personal mobility, especially in urban settings. Cars that are able to drive autonomously to pick up customers, take them to their destination, and then park themselves (or serve the next customer) can provide a mobility service that is almost as convenient as privately owned cars, with the sustainability of public transportation."

Another contribution from Frazzoli to autonomous vehicle technology is a mathematical model he developed with Carlo Ratti, a professor of the practice and director of the SENSEable City Lab in MIT’s Department of Urban Studies and Planning. The model plans for an autonomous, or “slot-based,” intersection” (Sl). These intersections remove the need for traffic lights by allowing autonomous vehicles, acting in concert as part of the Internet of things, to communicate with one another to ensure that each arrives at an intersection precisely when a “slot” required to pass through safely becomes available.

This process speeds up traffic flow by eliminating unnecessary stoppage, decreasing emissions and increasing efficiency. Frazzoli’s model demonstrates that it is possible to create a city without traffic lights, though such an achievement would require new innovation in other areas, for instance in developing ways for pedestrians and cyclists to move safely along with vehicles through Sl intersections.

Smart Incentives



The smartest of smart cities go even further than mechanical and systematic improvements, however, by helping their residents learn how to best conserve resources, including their own money. The large amounts of data gathered on transportation patterns in cities is helping researchers understand and develop incentives that encourage people to adapt their behaviors to a more efficient model, and to make more optimal choices, such as traveling during off peak hours.

MIT assistant professor and IDSS faculty member Jessika Trancik, in collaboration with Moshe Ben-Akiva, professor of civil and environmental engineering, is leading a large project that explores possibilities for helping people adapt their transportation behavior. The pair, along with several other MIT departments and a team at the University of Massachusetts at Amherst, are developing the Mobility Electronic Market for Optimized Travel, or MeMOT, a system in which consumers are rewarded — as they are in other areas of the marketplace — for optimal behavior. As Trancik remarks, “People make transportation choices based on their preferences and the information that they have. There is no question that access to information … affects personal transportation choices on a daily basis.”

By being given accurate, real-time information and feedback, consumers and residents are encouraged to exchange less efficient patterns of behavior for more efficient ones. In a smart city, where behaviors can be measured, data revealing actual behaviors and choices can also be more readily gathered, allowing for urban architects and engineers to learn which individual choices could be changed to improve overall quality of life and efficiency. Trancik remarks that’s “why models are important. Through modeling we can combine the most useful pieces of information in diverse data sets to provide a picture of the daily choices available to consumers of vehicles, drivers and travelers more generally.”

This sort of rapid responsiveness to readily available data, applied to individual choices about transport, energy, and other resources, could, in fact, be the very thing that finally closes the “open loop” of the energy markets, creating more efficient, reliable grids, something particularly necessary in the present and future age of “mega-cities.”

Smart Energy Systems



Energy is one of the markets in which it is simultaneously difficult and crucial for consumers to make good, informed choices. This is because typical energy markets are “closed,” as demand doesn’t vary according to price (although price fluctuates with demand). Energy market prices fluctuate with extreme frequency, and, since most customers have no access to this data in real-time, they are unprepared and unable to respond to these sorts of pricing markets. Customers instead tend to base energy choices on convenience rather than price, creating an open loop that allows energy companies to set price according to demand, but which doesn’t allow customers to respond to price. The normal feedback loop in these markets, however, is complicated by rapid price fluctuations.

When customers are given real-time pricing information in energy markets, as with “smart meters,” the consequences of good responses can, surprisingly, be devastating, according to research conducted by IDSS Director Munther Dahleh and others. If demand changes according to price fluctuation, which is the goal of most “smart technologies” that provide consumers with real-time data, rapid and erratic fluctuations in demand could result. Such fluctuations could cause a particularly dangerous situation in energy markets, which have “ramp constraints,” meaning that supply cannot easily keep up with rapidly fluctuating demand.

Dahleh and his collaborators, MIT research scientist Mardavij Roozbehani and Professor Sanjoy Mitter, are exploring just this nexus by using control theory (a branch of engineering and mathematics that studies how dynamic systems can be modified by feedback) to create a feedback loop for energy pricing that would allow for consumer pricing response, while mitigating excessive fluctuations in demand. Solving this problem could make for significantly better energy policy, contributing to more efficient and smarter cities that create less systemic risk to the grid, and make the grid considerably smarter.

Dahleh remarks that “the smart grid, through smart metering, will enable real-time demand shifting to cope with the uncertainty of renewable generation and to reduce the stress on the power grid. To realize this value, we are developing models and strategies to design incentive mechanisms, through pricing or availability of information, that will shape the consumer’s behavior in a fair and efficient manner.”

Future Promises

The smart cities of the not-too-distant future will themselves be feedback loops and smart grids comprised of interconnected, networked technologies that help people make informed choices based on efficiency, quality of life, and convenience. A sophisticated understanding of the data behind these complex networks will allow researchers to create continually improved systems that help people lead better, more efficient lives.

Although the prospect, and the reality, of smart cities does raise serious questions about cybersecurity, trust, and digital privacy, smart cities promise a great deal of improvement in the quality of life for their residents. The advances made by IDSS researchers, working across disciplines and domains, may even mean that tomorrow’s “mega cities” — which once threatened to drain natural resources, and cause massive congestion across systems — will instead be “mega smart cities,” fitted with highly-efficient interconnected systems that work together to offer residents a good, sustainable quality of life, and far more promising futures.

This article is part of a series highlighting major areas of research and innovation at MIT’s new Institute for Data, Systems, and Society (IDSS).

Courtesy of MIT News
[Read more]

5 Myths About The Future Of Artificial Intelligence (Debunked)

Posted On // Leave a Comment

The past decade has seen important advancements in computer science that enable software systems to compile and process new information to continually improve the way they function. This improved artificial intelligence is enabling computers to become an ever more powerful and valuable complement to human capabilities: improving medical diagnoses, weather prediction, supply-chain management, transportation, and even personal choices such as where to go on vacation or what styles of clothes to buy.


Although artificial intelligence has become commonplace -- most smartphones contain some version of AI, such as speech recognition -- the public still has a poor understanding of the technology. As a result, a diverse cast of critics, driven by fear of technology, opportunism, or ignorance, has jumped into the intellectual vacuum to warn policymakers that, sooner than we think, AI will produce a parade of horrible outcomes. Unfortunately, their voices have grown so loud that we are nearing a tipping point where their narratives may be accepted as truth, which would create a real risk that policymakers will decide to ratchet back the pace of progress.




With the White House convening a discussion later this week on the social and economic implications of artificial intelligence technologies, to be followed just days later by the 25th International Joint Conference on Artificial Intelligence, it is a good time to rebut these pervasive and pernicious myths.

Myth No. 1: AI will destroy most jobs.




Many now argue that AI will power a productivity explosion so great that it will destroy jobs faster than the economy can keep up, creating an unemployed underclass that will be dominated by an elite class of "machine owners." MIT professors Erik Brynjolfsson and Andrew McAfee write in their frequently cited book that workers are "losing the race against the machine, a fact reflected in today's employment statistics." These are not new predictions, and they are as wrong today as they have been in years past.

The apocalyptic views that AI will kill jobs suffer from two major errors. The first is that they vastly overestimate the capabilities of AI to replace humans. It is actually quite hard for technology, AI or otherwise, to eliminate jobs, as evidenced by the fact that U.S. productivity has been growing at a historically slow pace. And it is particularly hard to automate large numbers of jobs with AI, because virtually all AI is "narrow AI," designed to focus on doing one thing really well. So, in many occupations, the introduction of AI may not lead to job loss at all; it may instead increase output, quality, and innovation.

The second reason is that even if AI were more capable, there still would be ample job opportunities, because if jobs in one firm are reduced through higher productivity, then costs go down. These savings are recycled though lower prices or higher wages. This puts more money into the economy, and the money is then spent creating jobs in whatever industries supply the goods and services that people demand as their incomes go up. This is why, historically, there has been a negative relationship between productivity and unemployment rates.

Myth No. 2: AI will make us stupid.




Even beyond the unfounded fear that smart machines will take our jobs, some dystopians assert that AI will turn us into helpless automatons who are bound to become overly dependent on the machines and in so doing lose our own native skills -- so when the machines occasionally fail, we'll be ill-equipped to take back control. As author Nicholas Carr suggests, "Automation can take a toll on our work, our talents, and our lives."

To be sure, some skills may become less necessary as AI is able to handle routine tasks that humans used to do -- just as machines like the automobile made it unnecessary for most people to know how to ride a horse -- but it will open up new areas of skill. And the issue is not whether these systems won't make errors; it is whether on net they will make fewer errors than human-controlled activities. The answer is yes; they will make fewer errors -- otherwise they will not be used -- and that will be a boon to mankind.

Myth No. 3: AI will destroy our privacy.




If smart machines can crunch massive amounts of data, then surely they will destroy our privacy. Or so AI dystopians warn us. Reporter and author John Markoff writes, "This neo-Orwellian society presents a softer form of control. The internet offers unparalleled new freedoms while paradoxically extending control and surveillance far beyond what Orwell originally conceived. Every footstep and every utterance is now tracked and collected, if not by Big Brother then by a growing array of commercial 'Little Brothers'."

But there are several reasons why these opponents are wrong. First, while AI systems have the ability and even the need to collect and analyze more information, the threat to privacy is little greater than in non-AI systems, which already collect and analyze large amounts of information. Moreover, the rules that already govern data use and protect privacy today will cover data analyzed by AI, too.
In short, this is basically a policy question, not a technology question.

 If we don't want government agencies to collect certain data, then Congress can require that and courts enforce it. Whether agencies have or do not have machine-learning systems is irrelevant. In addition, many, if not most of the benefits of AI-enabled data analysis can be obtained without the need to risk disclosing personally identifiable information.

Myth No. 4: AI will enable bias and abuse.




Machine-learning systems are more complex than traditional software systems. It was relatively clear how the older rules-based expert systems made decisions. In contrast, machine-learning systems continuously adjust and improve based on experience. Some critics claim this level of complexity will result in "algorithmic bias" that promotes government and corporate abuse, whether unintentional or deliberate.

For example, Cathy O'Neil, author of the forthcoming book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, describes how machine learning algorithms are likely to be racist and sexist. Critics suggest that organizations will hide behind their algorithms and use the algorithms' complexity as a cover to justify exploitation, discrimination, or other types of unethical or damaging behavior.

It is certainly true that AI systems, like any technology, can be used unethically or irresponsibly. But those who resist AI based on this concern fail to recognize a key point: Machine-learning systems are not independent from their developers or the organizations using them. If an organization wants to systematically discriminate against certain groups, it'. need AI to do so.

 Furthermore, if an algorithmic system produces unintended and potentially discriminatory outcomes, it's not because the technology itself is malicious; it's because it simply follows instructions set by human decision making or, more often, relies on real-world data sets that may reflect bias. Finally, in most cases these systems are less biased than human decision making, where subconscious or overt biases permeate every aspect of society.

Myth No. 5: Smart machines will take over and potentially exterminate the human race.




Some argue that machines will become super-intelligent and decide they are better off without humans. Nick Bostrom, who has been called "a philosopher of remarkable influence," writes that a world with advanced AI would produce "economic miracles and technological awesomeness, with nobody there to benefit," like "a Disneyland without children," because the AI would first kill us all. Elon Musk, Bill Gates and Stephen Hawking's have also expressed concerns about "killer robots."

It's a sad commentary that the public has become so technophobic that we are even taking these sci-fi claims seriously. The view that smart machines will kill us overstates the pace of technological progress, particularly because the processing power of silicon computer chips is slowing down and progress in AI outside of deep learning is relatively modest. Moreover, machines and the human mind are completely different systems, and even major advances in computing are highly unlikely to produce a machine with humanity's intellectual capacity, imagination, or adaptability.

As MIT computer scientist Rodney Brooks puts it, "We generalize from performance to competence and grossly overestimate the capabilities of machines -- those of today and of the next few decades." Just as importantly, even if human-level intelligent machines could be built, which is unlikely, they will remain under the control of humans, because we would never build them unless they are largely safe, with the benefits outweighing the costs (just as we do with all technologies in the marketplace).

Author Bio

Robert D. Atkinson, Ph.D. - President, Information Technology and Innovation Foundation @RobAtkinsonITIF

Courtesy Of HuffingtonPost
[Read more]

Big Data What it is and Why It Matters

Posted On // Leave a Comment
Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.


Big Data History and Current Considerations



While the term “big data” is relatively new, the act of gathering and storing large amounts of information for eventual analysis is ages old. The concept gained momentum in the early 2000s when industry analyst Doug Laney articulated the now-mainstream definition of big data as the three Vs:

Volume. Organizations collect data from a variety of sources, including business transactions, social media and information from sensor or machine-to-machine data. In the past, storing it would’ve been a problem – but new technologies (such as Hadoop) have eased the burden.

Velocity. Data streams in at an unprecedented speed and must be dealt with in a timely manner. RFID tags, sensors and smart metering are driving the need to deal with torrents of data in near-real time.

Variety. Data comes in all types of formats – from structured, numeric data in traditional databases to unstructured text documents, email, video, audio, stock ticker data and financial transactions.

At SAS, we consider two additional dimensions when it comes to big data:

Variability. In addition to the increasing velocities and varieties of data, data flows can be highly inconsistent with periodic peaks. Is something trending in social media? Daily, seasonal and event-triggered peak data loads can be challenging to manage. Even more so with unstructured data.

Complexity. Today's data comes from multiple sources, which makes it difficult to link, match, cleanse and transform data across systems. However, it’s necessary to connect and correlate relationships, hierarchies and multiple data linkages or your data can quickly spiral out of control.

Big Data’s Big Potential



The amount of data that’s being created and stored on a global level is almost inconceivable, and it just keeps growing. That means there’s even more potential to glean key insights from business information – yet only a small percentage of data is actually analyzed. What does that mean for businesses? How can they make better use of the raw information that flows into their organizations every day?

Why Is Big Data Important?


The importance of big data doesn’t revolve around how much data you have, but what you do with it. You can take data from any source and analyze it to find answers that enable 1) cost reductions, 2) time reductions, 3) new product development and optimized offerings, and 4) smart decision making. When you combine big data with high-powered analytics, you can accomplish business-related tasks such as:


  • Determining root causes of failures, issues and defects in near-real time.
  • Generating coupons at the point of sale based on the customer’s buying habits.
  • Recalculating entire risk portfolios in minutes.
  • Detecting fraudulent behavior before it affects your organization.

Who Uses Big Data?


Big data affects organizations across practically every industry. See how each industry can benefit from this onslaught of information.



Banking

With large amounts of information streaming in from countless sources, banks are faced with finding new and innovative ways to manage big data. While it’s important to understand customers and boost their satisfaction, it’s equally important to minimize risk and fraud while maintaining regulatory compliance. Big data brings big insights, but it also requires financial institutions to stay one step ahead of the game with advanced analytics.

Education

Educators armed with data-driven insight can make a significant impact on school systems, students and curricula. By analyzing big data, they can identify at-risk students, make sure students are making adequate progress, and can implement a better system for evaluation and support of teachers and principals.

Government

When government agencies are able to harness and apply analytics to their big data, they gain significant ground when it comes to managing utilities, running agencies, dealing with traffic congestion or preventing crime. But while there are many advantages to big data, governments must also address issues of transparency and privacy.

Health Care

Patient records. Treatment plans. Prescription information. When it comes to health care, everything needs to be done quickly, accurately – and, in some cases, with enough transparency to satisfy stringent industry regulations. When big data is managed effectively, health care providers can uncover hidden insights that improve patient care.

Manufacturing

Armed with insight that big data can provide, manufacturers can boost quality and output while minimizing waste – processes that are key in today’s highly competitive market. More and more manufacturers are working in an analytics-based culture, which means they can solve problems faster and make more agile business decisions.

Retail

Customer relationship building is critical to the retail industry – and the best way to manage that is to manage big data. Retailers need to know the best way to market to customers, the most effective way to handle transactions, and the most strategic way to bring back lapsed business. Big data remains at the heart of all those things.

You can read more about specific case studies and insights here

Compiled By Marcel Masaga @magesa an unrelenting Big Data Advocate

Courtesy of SAS
[Read more]

How To Use - The Expert System AI Innovation To Solve Your Big Data Problems

Posted On // Leave a Comment
Summary

Even with all the newfangled systems and algorithms that are flooding the data science world, there's nothing wrong with using a tried-and-true solution that's been around for decades: the expert system. Don't let the simplicity of design fool you into thinking it's outmoded or ineffective, as the contrary is true.



The expert system is considered a data science dinosaur by some analysts, but this tried-and-true AI solution still holds up. Here are best practices to building an expert system.

I often come across data science teams that are fascinated by deep learning, compressive classification, and self-driving cars, and that are eager to employ the algorithm du jour. For instance, I was recently working with a large financial institution on increasing its cybersecurity and, before we even got started with basic monitoring, a data scientist on my team was talking about k-means clustering and neural networks.

We must always remember to understand the problem and opportunity first, and then apply the right system or algorithm. Sometimes self-learning neural networks may be the best alternative; however, sometimes you'll have to go with a classic: the expert system.

An expert system is a rule-based engine based on the collective wisdom of experts. It's one of the oldest innovations in artificial intelligence (AI), with its practical use dating back to the 1970s.

Expert systems are often joked about in data science circles as obsolete dinosaurs that are interesting but impractical for modern-day use. I strongly disagree—there's nothing in the advancements of AI that fully replace the functionality and utility of expert systems. Furthermore, since expert systems have been around for quite some time, there's a lot of history that you can leverage for best practices. Here are six of my best practices for using expert systems to get you started.




1: Gather requirements

One of the hardest parts about building an expert system is getting time with the actual experts. It's difficult enough to get time with any end users, but the experts you'll need for your project are very special end users that everybody wants to talk to. Before collecting requirements, engage management to confirm your time with the experts.

For instance, when I was working with a global transaction processing company, there were only a half-dozen or so people in the entire company that knew the guts of the transaction network. If you didn't get management to commit the experts' time, you aren't going to be talking to them for more than 15 minutes.

2: Perform analysis

Spend very little time on analysis. Resist the urge to perform qualitative analysis on expert interviews—it's not necessary.An expert system is designed to perform its own analysis. The hard work isn't so much in the analysis, but rather the setup and fine-tuning of your framework. In that respect, it's similar to a neural network. Your job is to tell the system how to think, and then let the system do the thinking for itself.

3: Design the framework

Design verbosity into your expert system framework. An expert system is comprised of two basic components: a knowledge base and an inference engine. The knowledge base stores facts about your domain under design, and the inference engine applies the AI equivalent of inductive (forward chaining) and deductive (backward chaining) reasoning to the facts in the knowledge base.

Both systems must be instrumented very well, allowing you to understand what the expert system is thinking. You need a very detailed view into what your expert system knows and how it's drawing its conclusions. Advanced systems employ more of a natural language interface—a best practice I support.

4: Develop the system

Development should be fast. Like analysis, if you're spending a lot of time here, you're doing something wrong. The only thing you should be developing is a framework (knowledge base and inference engine). Stay away from writing procedural code for now.

However, think ahead. Build interfaces where procedural code could take the place of framework inference. Although supplanting procedural code for framework-based inference is contrary to popular wisdom, it's a practical extension of your expert system once its rules have been thoroughly vetted. Procedural code affords you the opportunity of much faster execution, which is more practical for many applications (e.g., embedded systems).



5: Train the system

Don't underestimate the time, effort, and expert involvement required to properly train an expert system. I use the term train loosely—an expert system is not technically a learning system. But, what makes or breaks an expert system is its domain knowledge and how it makes inferences. Experts must be part of this process because they'll need to fine-tune the engine once it's seeded with the information collected during the requirements phase.

Here's where things get interesting. It's hard enough to get one expert to explain their process, let alone a group of experts to agree on the right process. It's worth it in the end, but diligence and patience will serve you well during this phase.

6: Improve the system

Retain your expert council for future reviews. Once your expert system has been deployed, it will be difficult to hold onto your experts for very long; they would need to periodically review real-world conclusions to ensure your system is still upholding its responsibilities as an expert. Get this commitment from them upfront. Use management like you did in the requirements phase—it's safe to say you'll need their eyeballs about once per quarter for at least a year while the expert system stabilizes. Make sure everyone's on board with this idea before you get started.

Conclusion

Even with all the newfangled systems and algorithms that are flooding the data science world, there's nothing wrong with using a tried-and-true solution that's been around for decades: the expert system. Don't let the simplicity of design fool you into thinking it's outmoded or ineffective, as the contrary is true.

As long as you secure the right experts, you can stand up an expert system in no time; meanwhile, other data scientists are still trying to wrap their brains around compressive classification. With the tips I've given you here and your own lessons learned, you'll be an expert systems pro before you know it.

TECHREPUBLIC
[Read more]