NASA Returns Hubble to Full Science Operations

NASA’s Hubble Space Telescope team recovered the Space Telescope Imaging Spectrograph on Monday, Dec. 6, and is now operating with all four active instruments collecting science. The team has still not detected any further synchronization message issues since monitoring began Nov. 1.

The team will continue work on developing and testing changes to instrument software that would allow them to conduct science operations even if they encounter several lost synchronization messages in the future. The first of these changes is scheduled to be installed on the Cosmic Origins Spectrograph in mid-December. The other instruments will receive similar updates in the coming months.

Hubble has been operating now for over 31 years, collecting ground-breaking science observations that have changed our fundamental understanding of the universe. With the launch of the Webb Telescope planned for later this month, NASA expects the two observatories will work together well into this decade, expanding our knowledge of the cosmos even further.

Nov. 29, 2021 — NASA Takes Another Step Toward Full Hubble Ops: Spectrograph Returns

The Hubble Space Telescope team recovered the Cosmic Origins Spectrograph instrument on Sunday, Nov. 28, moving the telescope further toward full science operations. Three of Hubble’s four active instruments are now collecting science data once again.

The team also continued work on developing and testing changes to instrument software that would allow them to conduct science operations even if they encounter several lost synchronization messages in the future. Those changes would first be installed on the Cosmic Origins Spectrograph once they’re completed and tested within a few weeks. Hubble’s other instruments would also receive similar changes. The team has not detected further synchronization message issues since monitoring began Nov. 1.

Nov. 22, 2021 — NASA Closer to Full Hubble Operations as Another Instrument Resumes Science

NASA continues bringing the Hubble Space Telescope back to normal science operations, most recently recovering the Wide Field Camera 3 instrument Sunday, Nov. 21. This camera will be the second of Hubble’s instruments, after the Advanced Camera for Surveys, to resume science after suspending the spacecraft’s observations Oct. 25. The Wide Field Camera 3's first science observation since the anomaly will be Nov. 23.

The team chose to restore the most heavily used Hubble instrument, the Wide Field Camera 3, which represents more than a third of the spacecraft’s observing time. Engineers also began preparing changes to the instrument parameters, while testing the changes on ground simulators. These changes would allow the instruments to handle several missed synchronization messages while continuing to operate normally if they occur in the future. These changes will first be applied to another instrument, the Cosmic Origins Spectrograph, to further protect its sensitive far-ultraviolet detector. It will take the team several weeks to complete the testing and upload the changes to the spacecraft.

Although the team has identified no further message losses since monitoring began Nov. 1, NASA is taking extra steps to keep the hardware safe in case the issue reoccurs. Investigation continues into the cause of the missed messages. The remaining Hubble instruments are still in safe mode and the rest of the spacecraft continues to operate as expected.

Nov. 16, 2021 — NASA’s Next Steps to Return Hubble Instruments to Normal Operation Status

The Hubble Space Telescope’s Advanced Camera for Surveys has continued collecting science data as NASA works to address the anomaly that started Oct 25. Missed synchronization messages halted science production at that time, and the team has identified no additional missed messages since monitoring began Nov. 1.

Hubble’s additional instruments have remained in safe mode while NASA takes steps to recover them to operational status. The rest of the telescope is operating as expected.

During the past week, the Hubble team has identified near-term changes that could be made to how the instruments monitor and respond to missed synchronization messages, as well as to how the payload computer monitors the instruments. This would allow science operations to continue even if several missed messages occur. The team has also continued analyzing the instrument flight software to verify that all possible solutions would be safe for the instruments.

This upcoming week, the team will begin to determine the order to recover the remaining instruments, including schedules for changing the instrument parameters before testing and developing the procedures. They also will test these changes to ensure they work as planned while continuing to isolate the root cause of the error.

The team expects it will take several weeks to complete these activities for the first instrument. The team has not yet determined which instrument would receive these changes first. In the meantime, they will start taking steps to recover Wide Field Camera 3 with no new changes next week, as was done with the Advanced Camera for Surveys, as an interim and low-risk step toward resuming normal science operations.

Nov. 8, 2021 — Hubble’s Advanced Camera for Surveys Instrument Resumes Science, Investigation Continue

The Hubble team successfully recovered the Advanced Camera for Surveys instrument Nov. 7. The instrument has started taking science observations once again. Hubble’s other instruments remain in safe mode while NASA continues investigating the lost synchronization messages first detected Oct. 23. The camera was selected as the first instrument to recover as it faces the fewest complications should a lost message occur.

Over the past week, the mission team has continued investigating the root cause of the synchronization issues and has seen no additional problems. The team will continue looking into possible short-term solutions this week and develop estimates for implementation. Once this occurs, the team will discuss returning the other instruments to operational status and resuming their science observations.

Nov. 4, 2021 — NASA Takes Additional Steps to Investigate Hubble Instruments in Safe Mode

NASA is continuing work to resolve an issue that has suspended science operations on the Hubble Space Telescope. The science instruments entered a safe mode configuration on Oct. 25 after detecting a loss of specific data synchronization messages.

The Hubble team is focusing its efforts to isolate the problem on hardware that commands the instruments and is part of the Science Instrument Command and Data Handling Unit. Specifically, the team is analyzing the circuitry of the Control Unit, which generates synchronization messages and passes them onto the instruments.

While analyzing the Control Unit, the team is working to identify potential workarounds for the issue. These include possible changes to instrument flight software that could check for these lost messages and compensate for them without putting the instruments into safe mode. These workarounds would first be verified using ground simulators to ensure they work as planned.

Over the weekend of Oct. 30, the team prepared to turn on parts of the Near Infrared Camera and Multi Object Spectrometer (NICMOS) instrument to collect data on this issue, allowing the team to determine how frequently this problem occurs. Installed in 1997, NICMOS has been inactive since 2010, when the Wide Field Camera 3 became operational. NICMOS allowed the team to use an instrument to collect information on these lost messages while keeping the active instruments off as a safety precaution. Since NICMOS was recovered on Nov. 1, no additional synchronization messages have been lost.

The team is now taking steps to recover Hubble’s Advanced Camera for Surveys (ACS) instrument from safe mode and start collecting science with that instrument at the beginning of next week. The team will make the decision on Sunday after analyzing the latest data. If a lost message is seen before then, the decision to activate ACS will also be revisited. The team is proceeding cautiously to ensure the safety of the instruments and avoid additional stresses on the hardware. Therefore, only ACS will be used in this capacity next week. ACS was selected as the first instrument to recover as it faces the fewest complications should a lost message occur.

Over the next week, the team will continue analyzing the Control Unit design diagrams and data associated with the lost messages to determine what may have caused this problem. They will also be looking into potential instrument software changes that could help address it. Once the team better understands the frequency of the problem and has determined the time needed to implement possible software changes, they will discuss a plan for returning the other instruments to science operations.

Nov. 1, 2021 — Hubble Instruments Remain in Safe Mode, NASA Team Investigating

NASA is continuing to investigate why the instruments in the Hubble Space Telescope recently went into safe mode configuration, suspending science operations. The instruments are healthy and will remain in safe mode while the mission team continues its investigation.

Hubble’s science instruments issued error codes at 1:46 a.m. EDT Oct. 23, indicating the loss of a specific synchronization message. This message provides timing information the instruments use to correctly respond to data requests and commands. The mission team reset the instruments, resuming science operations the following morning.

At 2:38 a.m. EDT, Oct. 25, the science instruments again issued error codes indicating multiple losses of synchronization messages. As a result, the science instruments autonomously entered safe mode states as programmed.

Mission team members are evaluating spacecraft data and system diagrams to better understand the synchronization issue and how to address it. They also are developing and testing procedures to collect additional data from the spacecraft. These activities are expected to take at least one week.

The rest of the spacecraft is operating as expected.