ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

An Interface for Biomedical Big Data Processing on the Tianhe-2 Supercomputer | JoVE Visualize

Home
Research Domains
Information And Computing Sciences

An Interface for Biomedical Big Data Processing on the Tianhe-2 Supercomputer

Xi Yang¹, Chengkun Wu², Kai Lu³

¹College of Computer, National University of Defense Technology, Changsha 410073, China. yangxi1016@nudt.edu.cn.

Molecules (Basel, Switzerland)|December 2, 2017

View abstract on PubMed

Summary

Orion simplifies big data processing on the Tianhe-2 supercomputer by enabling applications to run via a single command. This interface, leveraging high-performance computing (HPC), offers a practical and economical solution for big data analytics.

Area of Science:

Computer Science
Data Science
High-Performance Computing

Background:

Cloud computing and big data processing are increasingly integrated, utilizing frameworks like Hadoop and Spark.
The growth of high-performance computing (HPC) in China presents new opportunities for big data challenges.

Purpose of the Study:

To introduce Orion, a novel big data interface designed for the Tianhe-2 supercomputer.
To enable seamless execution of big data applications on Tianhe-2 with minimal user effort.

Main Methods:

Orion provides a single-command or shell script interface for launching big data applications.
It supports multiple users and tasks, employing an 'allocate-when-needed' resource management strategy.
Automated configuration minimizes the effort required to initiate applications.

Main Results:

Orion was tested using a large genomic dataset, demonstrating satisfactory performance on the Tianhe-2 supercomputer.
Existing Hadoop/Spark applications required minimal modifications to run with Orion.

Conclusions:

Orion offers a practical and economical solution for big data processing on supercomputing infrastructure.
The interface effectively bridges big data frameworks with HPC capabilities.

Distributed Computing And Systems Software

Cloud Computing

An Interface For Biomedical Big Data Processing On The Tianhe-2 Supercomputer

Keywords:

Hadoop Spark Tianhe-2 big data genomics big data