Big Data on ibm.com – The home for information on the complete IBM Big Data Platform, you can also find the free ebook (Harness the Power of Big Data), the Forrester Wave for Enterprise Hadoop Solutions, and more.

The Big Data Hub – This community includes fantastic conversations on a wide variety of formats, including blog posts, podcasts, videos, infographics, and more. Many of the contributors presented at IOD, and they include IBMers, IBM Champions, partners, analysts, consultants and more.

Big Data University – With over 50,000 registered students – and more signing up every day – Big Data University is a great way to dig into the details of big data. Big Data University offers more than 20 online courses and tutorials on topics like Hadoop, DB2, SQL and more.

The DeveloperWorks Big Data Community – This community is full of developer and DBA resources, tutorials and articles on big data technology and, specifically, the IBM Big Data Platform.

But that’s just what already exists. Follow your preferred social network for more as it develops:

Facebook: facebook.com/IBM big data

Twitter: @IBMbigdata

LinkedIn: IBM big data

Google+: IBM big data

YouTube: IBM big data

Also check out some of our most-viewed big data-related posts:
It’s a Big Data World! What Skills Do You Need?

Deepen the Business Value of Big Data: IOD Announcements

What Can Pop-Tarts Teach Us About Big Data?

Podcast with Nancy Kopp-Hensley: The IBM Big Data Platform for Analytics

Hadoop: An Important Piece of the Big Data Puzzle

Analyzing Streaming Data

What’s your favorite big data resource?

Jonas observed that as the systems we rely on and the data we collect continue to grow, the more important it is to “widen our observation space.” The more broadly we can expand our field of vision, the better the context we can build. And, the better the context we can build, the more relevant – and therefore more impactful – the insights we can achieve.

More signal, less noise.

I’ve taken Jonas’ advice to heart. Below are a few of my lingering thoughts about what we all learned in Las Vegas. They’ve been rattling around in my brain for a while; only recently have they formed into coherent thoughts. They’ve helped widen my own observation space; I hope they can do the same for you. Here goes:

Technology discussions aren’t about technology anymore. At least not entirely. There will always be a need to talk about storage and security. But IT discussions now are more likely to touch upon public policy, aesthetics and broad-based social and economic change. Irving Wladasky-Berger summed up it up during the What’s Next panel: “Technology becomes pervasive when people stop talking about specific devices and more about their large-scale effects.” Your cel phone was once just a phone. Now, it’s an entry point into a vast ecosystem of people, processes and data. When technology is everywhere, everything is open to the prospect of change, evolution, analytics and optimization.

We need a new way to understand its potential, and its potential effects. Geeks may make the world go ‘round, but philosophers and artists tell us what it all means. This is why Jason Silva was such an inspired choice to host this year’s event. His openness and enthusiasm for technology inspired many attendees; his education in philosophy and film helped him explain where it’s taking us in dramatic, compelling ways. Math needs the Humanities. And artists. And communicators who can explain the numbers in ways that drive people to make positive change.

We need better education about analytics and statistics. The drive to analyze and optimize will continue to expand as more data becomes available in more and more fields. Outcomes – whether in business or in public policy – will increasingly be determined and reported on as predictions and percentages. Cross-discipline collaboration and old-fashioned problem solving will be highly prized skills. It’s a positive trend, but as Nate Silver observed, it also demands a deeper understanding and a greater appreciation of what the data can and cannot do. At least not yet.

We all need to step up our game. The data – and hence, the potential – is out there for all of us to drive positive change, whether for our company, citizens, even our own health. It’s an unprecedented development in our history together on this planet. We owe it to each other to take advantage.

The jobs are out there. The worldwide shortage of data scientists was already common knowledge before HBR called it “the sexiest job of the 21st century.” As the world’s data volumes expand, so too will opportunities to develop what will be highly sought-after skills. Companies and governments are combing through their data for new opportunities to grow. The more quickly you can find them, or the greater the benefit, the more in demand you’ll be.

Reasons enough to Think Big, don’t you think?

Martin Wildberger presenting big data scenarios

If there was one takeaway from Information On Demand 2012, it was that the era of big data has arrived. Technology is available today to help us achieve the holy grail of Information Management: the ability to not only manage and store data but also integrate and analyze high volume, variety and velocity data in real time.

We can unleash the power of information with greater speed and efficiency while also detecting fraud and improving security. Seems too good to be true, right? Wrong!

One of the most memorable aspects of IOD 2012 was hearing from customers who are putting the InfoSphere technologies to the test and consequently solving daily business challenges. ConocoPhillips, Aetna, MoneyGram, Dillards, Nationwide, Citigroup, T-Mobile, Vestas Wind Systems and many others shared their big data journeys. In addition, demos of InfoSphere Guardium, InfoSphere Optim, InfoSphere Data Explorer and much more turned me into a believer.

During the Information Management keynote, we heard from Martin Wildberger, VP, WW Information Management Development, that InfoSphere closes the gap between big data challenges and opportunity by applying best practices around Information Integration and Governance to big data systems.

For example, with the announcement of InfoSphere Guardium v9.0, we can expand data security into big data environments such data warehouses, Hadoop-based systems and file shares. Another example, with InfoSphere Optim Data Privacy v9.1, we can protect data in big data systems with intelligent data masking techniques. Read more about these announcements in the October 18, 2012 IBM press release. Both of these technologies have a proven track record of delivering protection for relational data sources. These best practices have been expanded to embrace the new era of computing.

This message was also prevalent in the main tent keynote address by Steve Mills, Senior VP and Software Group Executive. Mills talked about how Information Integration and Governance is now more important than ever and is the foundation on which we should build all our big data projects.

With the InfoSphere Optim portfolio organizations can build common rules and policies across systems including privacy, retention, archive and retirement policies to ensure they are honored and enforced across the data lifecycle. InfoSphere Optim also provides a self services approach to governance spanning all roles and business units rather than being relegated to silos. InfoSphere Optim makes governance easy to design, execute, manage and easily extends to new sources of data.

Mills also commented that big data isn’t just about managing the data explosion: it’s about working smarter. InfoSphere Optim data lifecycle policies allow us to expand existing best practices and embrace a new era of computing. Don’t throw away filesystems and databases. Instead, extend the governance policies you have to address big data challenges across physical, virtual and cloud systems.

Mills commented that big data is a team sport shared across database experts, application owners, security professions, executives, partners and much more.

IBM has lived through massive paradigm shifts over its 100 year history. Now IBM is embracing the next generation bringing manageable and repeatable processes to harness and control big data. Mills left the audience with a fascinating story. If the US could eliminate the amount of fraud and waste in the federal government, we could retire the entire US debt in a decade.  This is truly a big data project. What’s yours?

Related reading:

New! ebook: Top Tips for Securing Big Data Environments

 New! ebook: Top 3 Myths about Big Data Security

Updated! Solution Sheet: InfoSphere Guardium

Enhanced! InfoSphere Optim Masking - IBM InfoSphere Optim Data Masking solution – Mask data on demand to protect privacy across the enterprise

Enhanced! InfoSphere Optim Data Growth

Enhanced! InfoSphere Optim for Warehouses

After catching up on some much-needed sleep, I settled into work Monday and found that several of my colleagues have been busy recapping their IOD experiences. Delaney Turner has a handy list of additional resources, including a reading list from Jason Silva.

But here’s what’s still rattling around in my head:

Making – and Trusting – Predictions

Nate Silver’s Wednesday keynote presentation was fascinating, as was the interview he did with Todd Watson and Scott Laningham (watch it here). Silver spoke about why and how we make predictions, and when we should trust such predictions. He stressed, “Collectively, we’re smarter than anyone is individually” – something we should remember as we all return to our respective jobs and challenges. As Silver explained, when solving problems, we should look at multiple theories and models rather than relying on a single indicator. I picked up his book while at IOD and can’t wait to dig in.

http://www.youtube.com/watch?v=Pu320CGrC8A

Analytics Are for Everyone

Saturday morning, I rode a train into Chicago. While rolling through a freight yard, I remembered the intriguing Information Management keynote on Monday. Kent Collins, Senior Database Architect with BNSF Railway (go to the 54:00 mark to hear Collins). BNSF has 220,000 active cars on the railroad across 32,000 miles in 28 states and two Canadian provinces. In 2010, BNSF hauled enough grain to make a year’s supply of bread for 900 million people and enough asphalt to lay a single lane road around the equator – four times.

While many may think of railroads as an old-school industry, BNSF is finding tremendous value in using predictive analytics to predict failures and

“We’ve got data everywhere,” Collins explained, noting they have over 1700 servers to track maintenance, weather, scheduling, inventory, safety, deliveries and more. These are mission critical to BNSF – as Collins noted, “The systems we put on PureScale can never go down… In testing, we did everything but pour water into the thing. We tried to kill it, and it wouldn’t die.”

Go to the 54:00 mark to hear Collins’ story

Freeing up resources to really dig into analytics and their implications has proved hugely valuable to BNSF. Collins continued, “If I can do it in my sleep, I want somebody or something else to do it. I don’t want to do the mundane. I want to do is help my business customers meet their objectives. I want to unlock the data and open up the environment.”

But the thing that really struck me was Collins’ ice cream analogy for PureScale. Before enjoying the benefits (sweetness or analytics), you must first buy all the components – the ice cream, bowls, scoops, etc. Then, you can either use some muscle or wait for it to soften a bit before scooping. But PureScale is more like an ice cream shop experience: everything you need is in one place. You can focus on customizing your cone exactly how you want it, and it’s ready to go.

Big Data Needs Context

This year’s IOD theme was “Think BIG,” and many of the sessions at least mentioned big data. But Jeff Jonas’ Wednesday morning talk reminded us that, “Just because you have a big pile of data doesn’t mean there’s gold in the hills” – and that the key is to understand the context, as well as what’s missing. As Jonas explained, “No one writes ‘bomb’ on the manifest!” You have to widen your observation space to make quality predictions. And that means understanding your data’s context.

Listen to Jonas’ explanation of G2 at 32:00

“Context accumulation is the future of big data,” Jonas stressed before revealing his new skunk works project, G2. Leveraging InfoSphere Streams and SPSS Modeler, G2 is a general purpose context accumulating engine. Basically, G2 helps discern whether entities are unique or the same. Watch Jonas’ demo and listen to his examples, starting at the 32-minute mark.

Jonas shared a couple of examples of G2 in action. He’s used G2 to do genealogy research with the 1880 Census, which has “very weak” data. He also worked with Singapore’s government to predict risk to the Malacca Strait, through which half the world’s oil supply passes.

Watch some of these snippets and tell me what resonates. Or, now that you’re back from IOD, which stories are you sharing with colleagues?

The 2012 edition of Information On Demand certainly left us all with a lot to think about, whether it was our Big Future, its Big Possibilities or simply the new mandate to Think Big.

No doubt you’re wondering how to share with your colleagues all the great things you saw and learned over the course of those three days. Naturally, the best way to determine your next steps would be to contact your IBM rep or Business Partner.

For my part, I’ve compiled a list of assets and sites related to the big themes of this year’s conference, namely: Big data, Smarter Analytics, IBM PureSystems and Watson.These are by no means exhaustive, but they should help you get started.

Also, I’ve recapped the news releases and included a list of suggested reading (both online and in book form) to help you orient yourselves toward this interconnected and intelligent future we’re all building together.

Finally, you can always review our rich store of event video content on our Livestream channel. Review and revisist the opening general sessions, track keynotes and a long list of Scott and Todd’s interviews with keynote speakers and IBM executives.

Remember, you can always do a deep dive into any IBM Software capability on IBM.com.

Big Data: Manage and leverage volume, variety and velocity for maximum business value

• Visit The Big Data Hub for IBM customer references, blog posts from our community of big data thought leaders, videos and special offers.
• Download the new IBM/Oxford study, Analytics: The real-world use of big data
• Download our new eBook, Harness the Power of Big Data: The IBM Big Data Platform
• Follow IBMBigdata on Twitter, check out our big data Facebook page and join our Big Data Group on LinkedIn.

Smarter Analytics:Turn information into insight, and insight into outcomes

• Visit our Smarter Analytics Web site to explore the full breadth of our analytics solutions
• Analyze your own analytics with our new online self-assessment tool.
• Download a free copy of IBM Cognos Insight from our new Analytics Zone
• Visit us on Facebook or follow us on Twitter for the latest blog posts, updates and offers

IBM PureSystems: Simplified experience, built-in expertise and integration by design

• Download our white paper, “Expert Integrated Systems: A Next-Generation Computing Platform
• Get started with the free cloud-based trial and the pattern development kit.
• Get an accurate picture of the benefits and TCO savings by consolidating with PureFlex System
• Follow IBM PureSystems on Twitter

Watson: Ushering in a new era of computing

• Download our White Paper, “Cognitive Systems Redefine Business Potential”
• Download the IDC Report, “IBM’s Watson: From Wining Games to Saving Lives”

Press Releases

• IBM Helps Organizations Tackle Industry-Specific Big Data Challenges
• Made in IBM labs : Made in IBM Labs: New IBM Analytics Software Helps Doctors Transition to Predictive Healthcare
• IBM Announces Student Winners of Watson Case Competition from Cornell University

Books to read (from host Jason Silva’s bookshelf)

• What Technology Wants, Kevin Kelly
• The Singularity is Near, by Ray Kurzweil
• Where Good Ideas Come From, by Steven Johnson

Thursday ended a great journey for all the attendees and participants, and it is clear what an incredible and energy-filled week Information On Demand certainly was. One of the highlights from Thursday was the “Next Lunch” which featured three of IBM’s technical gurus, allowing for open discussion on the future of technology.

Also, Andrew Robinson shared several more photos from the conference illustrating all the “Big” opportunities the conference delivered. Richard Lee shared details from his final day at the conference and the key take aways and learnings of his week. Robinson also discussed all the common themes he saw during the week in Vegas in his summary of Thursday. Lastly, Trevor Eddols explains the many announcements and product news released throughout the conference.

Clearly this was a week full of opportunities, news, and ideas to get attendees to start to “Think Big.” Now that the conference has come to an end, what comes next?  Below is one of my favorite highlight videos from the week, focused on the closing general session – “Big Future” featuring Jeff Jonas and Nate Silver. The video includes great insights from the session as well as audience reaction.

http://www.youtube.com/watch?v=S2uFVZ3isO4

For many more highlights, videos, etc. be sure to check out the Information On Demand YouTube channel.

Some of the best from late Tuesday and Wednesday:

Sandy Sherrill mused on connections, mappings and a “rhythmic cascade” of learning at the conference, including a recap of Steve Mills’ general session, a best practices article on IMS tools, and tidbits from two customer/Champion sessions. Can someone tell Sandy where Farhana Alarakhiya bought her shoes?

Mario Daigle shares “that singular moment” that he’ll remember from IOD, when Cognos suddenly clicked for someone.

Susan Visser, who coordinated all the bookstore work and book signings ,released the IOD best seller list.

The Women in Technology panel was so well attended that I couldn’t get in! Luckily, both Darice Bailer and Susan Visser were there and willing to share what they heard.

Several of our Social Influencers have posted recaps, including:

Andrew Robinson: Day 3 at IOD – lots of learning #ibmiod
Andrew Robinson:  IOD- What have I heard/learned so far #ibmiod
Andrew Robinson: Day 4 – Wednesday – at IOD #ibmiod
Andrew Robinson: Interesting IOD Facts/Quotes #ibmiod
Tom Griggs: Perficient Wins Integration Excellence and Business Analytics Awards at IOD 2012
Richard Lee: Information on Demand_Tuesday 10/23/2012
Richard Lee: Information on Demand_Wednesday 10/24/2012
Paul Mendelson, aka “Cognos Paul”: IOD 2012- Sunday
Paul Mendelson, aka “Cognos Paul”: #iod 2012 Monday
Paul Mendelson, aka “Cognos Paul”: #ibmiod Tuesday: Exclusive sneak peek at Motio’s newest upcoming product
Paul Mendelson, aka “Cognos Paul”: #ibmiod Wednesday

And of course, Todd Watson has been busy attending sessions and grabbing the best soundbites. Watson’s Next Job recapped Craig Rhinehart’s session about how IBM Watson is helping IBM Patient Care solve a huge problem. According to Rhinehart, 30 cents of every healthcare dollar in the US is wasted.  Todd also got a chance to sit down with Nate Silver, and includes a really eye-opening interview.

Katrina Reed again offers a fantastic summary.

We’ll have additional posts in the coming days as new videos become available. Plus, several bloggers will be writing their thoughts about the conference after they’ve had a chance to reflect on all they’ve learned.

The discussion was very interesting, and I’ve summarized a few of the points that were covered.

Point 1: How can women be promoted as often as men? Women tend not to ask for promotions or raises. You should ask! The worst that could happen is you get the answer “no.” I recently read the book Ask For It: How Women Can Use the Power of Negotiation to Get What They Really Want that provides helpful advice on how to negotiate salary, promotions and much more.

Point 2: Don’t be afraid to make a statement. Make your voice be heard at meetings by giving your opinion. If you are a leader, motivate those on your team to provide their opinion. Employee – work for people who inspire you to be the best you can be. Parents – teach your children that everything is within their reach if they put the work in. A book that I liked on this topic is Mindset: The New Psychology of Success.

Point 3: What does success look like? People may not think the same way as you and you need to clarify with those around you as to what inspires them and what success means to them. For some it may be opportunity, while for others it may be autonomy, or even money. Figure out what motivates you and communicate this to your leaders. I suggest the book Making it Big in Software: Get the Job. Work the Org. Become Great to help you define what success means to you.

Point 4: Ask yourself: “If you knew you weren’t going to get fired, would you do anything differently?” Challenge the status quo and operate without fear. Take risks and don’t hold back.

Women tend to “self select out,” meaning they will not apply for a job that they don’t think they are qualified for, even if they are.

Point 5: “Occupy your space.” Own it all. Don’t rent your space. Create an impact and value in everything you do. What motivates you? Opportunity or Obligation. I suggest you watch this Ted.com video: Sheryl Sandberg: Why we have too few women leaders

Point 6: Is there a double standard for women who are assertive? Are they perceived as aggressive if they speak up? No, performance is the answer. Focus on creating value and on delivery. If you’re not being heard, you may need to work on this skill. Step outside your comfort zone and make your opinion known since it needs to be heard as well. If you’re leading, call on the shy ones on your team to help them voice their opinion. Make space at the table for people with a shy style. Here’s an article that touches on the topic: Are You Aggressive or Assertive? Frantic or Focused?

Question 7: Can you have work-life balance and grow in the ranks? Answer: There is no such thing! Instead think about “work life integration.” Work is part of your life. Some days there’s more family and some days there’s more work. Balance and success mean different things for different people. Strive to be happy, but don’t let fear hold you back. Also, think of balance over the course of your life / career instead of in a 24-hour time period. Look at the balance over your career… over the course of your life. If you’re starting to feel guilty about your choices, step back and look at your successes…family, career, etc; and give yourself a break! A great article on this subject: Why Women Still Can’t Have It All.

Point 8: Are there career obstacles unique to our gender? Don’t focus on WHY the obstacle arose, and don’t get bent out of shape if it did arise because of gender. If you focus on this, you will make gender the obstacle.

Point 9: As a manager, how do you develop employees? Make sure you know what your employee wants and be prepared to lose good employees. Relationships are critically important in a career like IBM. Stay in touch with those who leave. They may come back to work for you, or become a business partner, or more.

Point 10: Be credible from the start, especially as a leader. Keep it about the business. Be in an authority position and stay there. Be consistent and don’t handle things emotionally.

A reception followed this awesome discussion, allowing us to meet and network with the many women who attended IOD this year.

You might also enjoy Darice Bailer’s report of this panel discussion. 

Sandy Ovard has been working as Business Partner liaison to Lab Services and has helped us navigate the increasingly complex Lab Services partner engagement process. As we depend on Lab Services for a portion of our services business, Sandy has had the frequently unpleasant job of soliciting Business Partner feedback and addressing issues. Her patience and consistent willingness to pitch in will be missed.

Holly Hayes is one of the SVL development superstars who has worked with a number of technologies. Holly is one of the few “surf and turf” people who fully understand the most technical of technical subjects and has the communication skills to explain them to those non-techies, like me, who need analogies and pictures to understand. I am have been surprised – and amazed – on more than one occasion when I hear about a customer that struggled with a new product implementation and hear, “and then I got Holly involved,” and invariably, things smooth out.

Finally, David Tolleson. David has been the replication and federation Product Manager and a tireless promoter of these solutions. David has blogged, responded to endless emails, and enthusiastically supported a niche technology that enables some of the world’s largest IT shops to meet those 24X7 uptime SLA’s. He has worked with a team that is not only technically capable, but fun to work with.

Good luck to you all!

Ask for your next promotion. Believe that you’re just as good as the guy next to you. Get off the bench and onto the basketball court, taking shots to solve company problems. Make an impact. You have an opportunity and an obligation for greatness.

These were just some of the golden nuggets of advice offered by IBM’s Alyse Passarelli and a dynamic panel of technology leaders Passarelli assembled for Wednesday night’s annual Women in Technology panel on “Taking Charge of Your Career.”

Passarelli (IBM Vice President, Information Management Sales Software Group) moderated, the panel and the very lively discussion with IBM’s Deidre Paknad,  Vice President of Information Lifecycle Governance Solutions Business, Glenda Crisp, Vice President and CIO of Corporate Segment Technology Solutions at TD Bank Group, Kecha Mitchell, Vice President, Enterprise Information Management – Delivery Services at First Tennessee Bank, and Martin J. Wildberger, IBM Vice President, Worldwide Information Management Development.

The room was packed, and yet the women kept streaming in. Here are some ways women can take charge of their careers.

How do you get your next promotion?

Ask! “I’ve been married for 15 years and my husband can’t read my mind, so why do I think my boss could read my mind?” Paknad said, as the audience laughed. Men aren’t very observant in that matter, Mitchell chimed in. So ask. All your boss can do is say no.

How do you step outside your comfort zone and take risks to get ahead in your career?

Mitchell said you can do anything you want to do. And, don’t listen to the naysayers who say you can’t. There will be lots of people who will tell you all the reasons why you’ll never make it. Just keep going.

Paknad said she started three very successful companies. “I get to grow my own destiny,” Paknad said. She doesn’t have to ask anyone for a promotion, either. And, if you’re scared that you don’t know enough to forge ahead, push that thought right out of your mind. You probably know a whole lot more than the people to your left or your right, said Passarelli.

“Occupy your space,” Passarelli said. “It’s your job,” Paknad said. “Don’t rent it. Own it. You have the opportunity for greatness, and you have the obligation for greatness.”

Crisp said she’s always surprised when there’s a job posting at TD Bank and guys will call her up for the job but no women! Why? Women think they can’t meet all ten qualifications that are listed on the posting. That doesn’t happen with men. Men check off two boxes and say they’re qualified. Now women have to skip the negative talk and think like a man. If he can do it, you can do it. “That’s what the men are saying,” Crisp said.

And Wildberger encouraged women to have a frank discussion with their boss. Don’t ask how you’re doing, because if the boss tells you you’re doing a good job, well, everyone else is probably doing a good job, too. Ask a more pointed question. Ask where you’re ranked. How you stack up against the others. Ask, “Am I one of your top three candidates?” And, if the boss says you’re not, ask “What can I do to be one of your top three candidates?”

How can you make an impact in your life and in your career?

By definition, that mean’s change, Paknad said. If you keep on doing things the way they’ve always been done, you won’t move forward. “It’s a treadmill.”

Mitchell encouraged women to challenge the status quo in their careers. Don’t follow everyone around you who are doing things they way they’ve always been done because that doesn’t mean it’s the right way. Mitchell has been replacing a disjointed information infrastructure at First Tennessee Bank. “I’m always challenging the status quo,” said Mitchell. “That’s the way you want to go.”

“Put yourself in the middle of the intersection and say, `Okay, I’m going to do something,” Wildberger said.

How can women be assertive without being nasty?

“My answer is performance,” Paknad said. At the end of the day, Mitchell said she’s focused on delivery. She wants to be part of the solution getting the ball rolling down the field and fixing the company’s problem.

How can women balance being productive at work and home?

There’s no magic secret here, said Wildberger. Everyone is working long hours in IT these days, and everyone is struggling with this work-life balance. But technology can help. In fact, Wilberger will take his BlackBerry with him to watch his son’s games. Or e-mail his son an answer to a homework question from Singapore.

And balance isn’t about a 24-hour period, or a 72-hour period. It’s about the long-haul. Don’t think you have to balance your work and family every day, because it will balance out in the long-term and in the end you will raise some pretty phenomenal kids. So, when things are a little out of balance, give yourself a break.

Susan Visser also attended the panel  - read her perspective.