return to grid print    Jump to Issue:
Statusresolved
Assigned Torwiseman
TypeWebsite Issue
AreaGenetics Services
Priority3
Milestone 
Opened2013-09-01 by rwiseman
Changed2013-09-03 by Salendra Singh
Resolved2013-09-03 by Salendra Singh
ResolutionFixed
Closed

2013-09-01 rwiseman
Title»MiSeq data transfer to LabKey database (Continued)
Assigned To»gstarrett
Type»Website Issue
Area»Genetics Services
Priority»3
Today's MiSeq run (#122 or #604 depending on which numbering system you use) still hasn't completed it's import into LabKey ~12hrs after the run finished so that we can get to the binned FASTQ files. This looks like the same behavior that we experienced with our last run - all of the metrics data is there when you click on the "file browser" tab under "view data" for MiSeq 122 but there's nothing under the "sample data" tab for reads associated with each of the 188 different barcodes/indexs/MIDs. The FASTQ files were binned on the MiSeq in the appropriate output folder on the MiSeq itself around 10AM this morning so the data is processed. Under the "Data Pipeline" tab it says "importing reads" was created at "2013-09-01 12:59" but it must be stuck again since it shouldn't take >9hrs for the FASTQ files to transfer. Do we need to restart LabKey again?

2013-09-03 Salendra Singh
Assigned Togstarrett»Salendra Singh

2013-09-03 Salendra Singh
resolve as Fixed
Statusopen»resolved
Assigned ToSalendra Singh»rwiseman
Today's MiSeq run (#122 or #604 depending on which numbering system you use) still hasn't completed it's import into LabKey ~12hrs after the run finished so that we can get to the binned FASTQ files. This looks like the same behavior that we experienced with our last run - all of the metrics data is there when you click on the "file browser" tab under "view data" for MiSeq 122 but there's nothing under the "sample data" tab for reads associated with each of the 188 different barcodes/indexs/MIDs. The FASTQ files were binned on the MiSeq in the appropriate output folder on the MiSeq itself around 10AM this morning so the data is processed. Under the "Data Pipeline" tab it says "importing reads" was created at "2013-09-01 12:59" but it must be stuck again since it shouldn't take >9hrs for the FASTQ files to transfer. Do we need to restart LabKey again?


Diagnosis : All scripts working fine. Server processed data very slowly( Usually its 12500 reads/sec But specifically for this run and Manual import of Run No 97 it went down 210 reads/ sec( 12500 reads in a minute) . It may be because of excessive load on the server .

For Further Consideration this is the timeline

Data Transfer from MiSeq to Bonobo

[Sun Sep 1 12:30:01 2013] Copying MiSeq run directory, 130830_M01472_0069_000000000-A5R6E.
[Sun Sep 1 12:52:14 2013] Copying complete.
[Sun Sep 1 13:00:01 2013] Started Data Processing

Data Processing at Bonobo

Sun Sep 1 13:00:02 2013] Making run directory miseq122 and subdirectories
[Sun Sep 1 13:00:02 2013] Copying files from /media/RAID6-ext4/miseq_data/MiSeqOutput/130830_M01472_0069_000000000-A5R6E/Data/Intensities/BaseCalls to /media/RAID6-ext4/miseq_data/uploaded_runs/miseq122/BaseCalls
[Sun Sep 1 13:00:26 2013] Copying files from /media/RAID6-ext4/miseq_data/MiSeqOutput/130830_M01472_0069_000000000-A5R6E to /media/RAID6-ext4/miseq_data/uploaded_runs/miseq122
[Sun Sep 1 13:00:26 2013] Copying files from /media/RAID6-ext4/miseq_data/MiSeqOutput/130830_M01472_0069_000000000-A5R6E/InterOp to /media/RAID6-ext4/miseq_data/uploaded_runs/miseq122/InterOp
[Sun Sep 1 13:00:26 2013] Parsing /media/RAID6-ext4/miseq_data/uploaded_runs/miseq122/SampleSheet.csv header and uploading data into LabKey
[Sun Sep 1 13:00:26 2013] Starting import of Illumina Reads in LabKey
[Sun Sep 1 13:00:27 2013] Archiving and compressing raw run data directory
[Sun Sep 1 13:08:18 2013] Run processing complete

Data Processing on LabKey

Data Processing Started : 01 Sep 2013 12:59:35,055
Data Processsing Ended : 03 Sep 2013 09:14:03,990