The Briefing Room

General Category => Science, Technology and Knowledge => Space => Topic started by: Elderberry on January 24, 2019, 02:12:35 am

Title: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main camera
Post by: Elderberry on January 24, 2019, 02:12:35 am
Daily Mail By Mark Prigg 1/15/2019

    Premier camera was shut down because of a hardware problem

    Hubble's three other science instruments are still working
 
    Wide field camera was installed by spacewalking astronauts in 2009


It may be the classic response from IT departments around the world, but turning it off and on again really does work - even for the Hubble Space Telescope.

NASA has confirmed that resetting the aging telescope's main camera appears to have fixed an issue that left it offline.   

Hubble Space Telescope’s Wide Field Camera 3 instrument, suspended operations on Tuesday, Jan. 8th as a result.

Shortly after noon EST on Jan. 8, software installed on the Wide Field Camera 3 detected that some voltage levels within the instrument were out of the predefined range, NASA said.

The instrument autonomously suspended its operations as a safety precaution.

Upon further investigation, the voltage levels appeared to be within normal range, and all other telemetry within those circuits also contained erroneous values indicating that this was a telemetry issue and not a power supply issue.

After resetting the telemetry circuits and associated boards, additional engineering data were collected and the instrument was brought back to operations. 

Assuming that all tests work as planned, it is expected that the Wide Field Camera 3 will start to collect science images again by the end of the week.

Hubble's three other science instruments are still working fine, with celestial observations continuing.

More: https://www.dailymail.co.uk/sciencetech/article-6596513/Switching-DOES-work-NASA-reveals-reset-fixed-Hubble-Space-telescope.html (https://www.dailymail.co.uk/sciencetech/article-6596513/Switching-DOES-work-NASA-reveals-reset-fixed-Hubble-Space-telescope.html)
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: bigheadfred on January 24, 2019, 02:48:10 am
I wonder what they were looking at when it went offline?

Not saying it was aliens, but it was aliens.

Anytime NASA gets some good clear pictures of aliens they have an equipment malfunction.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: ConstitutionRose on January 24, 2019, 11:44:47 am
No reference to aliens here.  I was an IT tech at a nuclear power plant.  Walked by a group of engineers who were sweating over a new piece of technology from GE.  They were on the phone with GE and had been at it for hours.  I suggested they reboot it.  They chided me a little but rebooted it and wonder of wonders it worked.  Everything mechanical needs a reboot sometimes. 
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: bigheadfred on January 24, 2019, 01:30:45 pm
No reference to aliens here.  I was an IT tech at a nuclear power plant.  Walked by a group of engineers who were sweating over a new piece of technology from GE.  They were on the phone with GE and had been at it for hours.  I suggested they reboot it.  They chided me a little but rebooted it and wonder of wonders it worked.  Everything mechanical needs a reboot sometimes.

Everything needs a reboot. Humans call it sleep. But essentially it is the same thing. Turn off your conscious mind, wait ten minutes, turn it back on and everything is refreshed and works better.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Elderberry on January 24, 2019, 02:14:10 pm
Everything needs a reboot. Humans call it sleep. But essentially it is the same thing. Turn off your conscious mind, wait ten minutes, turn it back on and everything is refreshed and works better.

Maybe ten minutes works, I don't know. Naps don't work for me. I wake up groggy as hell. I have found though if I just have to get some rest and can't sleep, just resting with my eyes closed is very refreshing.

But that's for people. There is a lot of equipment that doesn't react well to being power cycled. Reboot yes. Power cycle no.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: bigheadfred on January 24, 2019, 02:37:03 pm
Maybe ten minutes works, I don't know. Naps don't work for me. I wake up groggy as hell. I have found though if I just have to get some rest and can't sleep, just resting with my eyes closed is very refreshing.

But that's for people. There is a lot of equipment that doesn't react well to being power cycled. Reboot yes. Power cycle no.

The ten minute remark is merely some reference I've seen for certain things. Like this laptop. Take the battery out, wait ten minutes.

I think I understand the mechanical part. Better to keep a diesel or electric motor running than repeated starts.

I don't do well with naps, either. Same thing if I go back to bed in the morning after getting up. In reality that should count as a nap. The effect on me is the same--groggy.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Elderberry on January 24, 2019, 03:59:00 pm
I think I understand the mechanical part. Better to keep a diesel or electric motor running than repeated starts.

Electrical/electronic equipment can fail on shutdown/restarts. In someone's infinite wisdom, they mounted an emergency power shutdown button in JSC's Bldg30 server room by the exit door on the hinge side. It got hit once by the cleaning lady's mop handle. Another time it got pushed by someone thinking they were hitting the door unlock button that was at the same height on the other side of the door. They put a guard on the power shutoff button after that. There was always some equipment that failed to restart and had to be replaced.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: dfwgator on January 24, 2019, 04:04:13 pm
(https://media.giphy.com/media/DUtVdGeIU8lmo/giphy.gif)
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: ConstitutionRose on January 25, 2019, 12:47:39 pm
Electrical/electronic equipment can fail on shutdown/restarts. In someone's infinite wisdom, they mounted an emergency power shutdown button in JSC's Bldg30 server room by the exit door on the hinge side. It got hit once by the cleaning lady's mop handle. Another time it got pushed by someone thinking they were hitting the door unlock button that was at the same height on the other side of the door. They put a guard on the power shutoff button after that. There was always some equipment that failed to restart and had to be replaced.

In my world its power outages and lightening strikes.  In one of our storms this past fall power cycled on and off repeatedly.  There were a lot of older computers that just refused to restart at some point.  And a lot of people who didn't listen to my advice to buy UPS equipment that were sorry.  Most modern computer equipment handles power cycles pretty well (as a one off).   
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: catfish1957 on January 25, 2019, 01:30:09 pm
No reference to aliens here.  I was an IT tech at a nuclear power plant.  Walked by a group of engineers who were sweating over a new piece of technology from GE.  They were on the phone with GE and had been at it for hours.  I suggested they reboot it.  They chided me a little but rebooted it and wonder of wonders it worked.  Everything mechanical needs a reboot sometimes.

Not really related, but back in the early '90's I was having PC problems, and I called our best IS person in the plant.

True story:  He turned my 386 off.  Waited 10 seconds, and whacked it on the side.  Machine worked fine, though he did tell me, that it took a certain level of whacking to get desired result.   :silly:
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: bigheadfred on January 25, 2019, 02:44:12 pm
Not really related, but back in the early '90's I was having PC problems, and I called our best IS person in the plant.

True story:  He turned my 386 off.  Waited 10 seconds, and whacked it on the side.  Machine worked fine, though he did tell me, that it took a certain level of whacking to get desired result.   :silly:

Reminds me of the cowboy poem about the cross-eyed bull.

Sorry... (http://www.youtube.com/watch?v=xEhocSfb9cg#)
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Elderberry on January 27, 2019, 03:32:38 am
Not really related, but back in the early '90's I was having PC problems, and I called our best IS person in the plant.

True story:  He turned my 386 off.  Waited 10 seconds, and whacked it on the side.  Machine worked fine, though he did tell me, that it took a certain level of whacking to get desired result.   :silly:

While in the Navy, there was a radar repeater that had an intermittent problem. Whenever it went spastic, the call would go out to Roos. He was the only one of us that was uninhibited enough to kick the hell out of it. The problem would then go away for awhile. Then one idle period Roos fixed it for good. He must of hurt his foot or just got tired of kicking it.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: InHeavenThereIsNoBeer on January 27, 2019, 03:40:25 am
Not really related, but back in the early '90's I was having PC problems, and I called our best IS person in the plant.

True story:  He turned my 386 off.  Waited 10 seconds, and whacked it on the side.  Machine worked fine, though he did tell me, that it took a certain level of whacking to get desired result.   :silly:

You don't hit them when they're down.  You hit them when they are up and on the network so their friends get the message as well.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Bigun on January 27, 2019, 03:40:38 am
While in the Navy, there was a radar repeater that had an intermittent problem. Whenever it went spastic, the call would go out to Roos. He was the only one of us that was uninhibited enough to kick the hell out of it. The problem would then go away for awhile. Then one idle perioda Roos fixed it for good. He must of hurt his foot or just got tired of kicking it.

My story is very similar but involved a radio transmitter that would just shut down for no reason. Long story short,  after many continuous hours of trying to track down the problem and fix it, I picked up the straw broom that was nearby and knocked the hell out of it.  Still working as far as I know.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: InHeavenThereIsNoBeer on January 27, 2019, 03:49:10 am
I took Elementary Numerical Analysis around the time the Patriot missles failed to knock down a SCUD which ended up hitting one of our barracks (Bahrain?).  Turns out the problem is that they hadn't been rebooted.

[The gist of it is they were using floating point to count time very accurately.  Due to the way FP is implemented, if you keep adding tiny little numbers to each other eventually the gap between what can be represented will be larger than what you are trying to add -- effectively, time stopped]
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Elderberry on January 27, 2019, 04:16:19 am
It always amazed me that they would use double precision floating point to process data from an 8 bit A/D.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: bigheadfred on January 27, 2019, 04:22:06 am
It always amazed me that they would use double precision floating point to process data from an 8 bit A/D.

Since I don't know I'll just ask. Was that overkill?
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Elderberry on January 27, 2019, 04:27:02 am
I took Elementary Numerical Analysis around the time the Patriot missles failed to knock down a SCUD which ended up hitting one of our barracks (Bahrain?).  Turns out the problem is that they hadn't been rebooted.

[The gist of it is they were using floating point to count time very accurately.  Due to the way FP is implemented, if you keep adding tiny little numbers to each other eventually the gap between what can be represented will be larger than what you are trying to add -- effectively, time stopped]

We had a similar problem metering Space Shuttle payload data to the payload users. Over time the metering would get worse and worse. I found that the best solution was to do no ground metering adjustment of the payload data at all. Just  depend on the metering from the Shuttle. And reboot the ground telemetry processors once a day.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Elderberry on January 27, 2019, 04:27:57 am
Since I don't know I'll just ask. Was that overkill?

64 bit words to handle 8 bits of data.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: corbe on January 27, 2019, 04:32:40 am
(https://i2.kym-cdn.com/photos/images/newsfeed/000/176/261/Windows_9X_BSOD.png)
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Elderberry on January 27, 2019, 04:39:32 am
We used to send some wild Security Violation Warning Pop-Ups to the newbies machines.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: bigheadfred on January 27, 2019, 04:40:48 am
64 bit words to handle 8 bits of data.

Thanks.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: DB on January 27, 2019, 04:43:44 am
Electrical/electronic equipment can fail on shutdown/restarts. In someone's infinite wisdom, they mounted an emergency power shutdown button in JSC's Bldg30 server room by the exit door on the hinge side. It got hit once by the cleaning lady's mop handle. Another time it got pushed by someone thinking they were hitting the door unlock button that was at the same height on the other side of the door. They put a guard on the power shutoff button after that. There was always some equipment that failed to restart and had to be replaced.

Yep, electronics that has been on for years may not come back on with a power cycle. I've seen it many times. It is something you don't want to do unless you've run out of alternatives (for long term constantly powered equipment). In addition you want to do as much trouble shooting as you can before power cycling to try to determine the cause. Otherwise if it works when repowered it will happen again because the root problem wasn't fixed.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: DB on January 27, 2019, 04:48:06 am
I took Elementary Numerical Analysis around the time the Patriot missles failed to knock down a SCUD which ended up hitting one of our barracks (Bahrain?).  Turns out the problem is that they hadn't been rebooted.

[The gist of it is they were using floating point to count time very accurately.  Due to the way FP is implemented, if you keep adding tiny little numbers to each other eventually the gap between what can be represented will be larger than what you are trying to add -- effectively, time stopped]

Floating point has to be handled with care... Small differences between big numbers goes wrong fast...
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: Elderberry on January 27, 2019, 05:05:21 am
Yep, electronics that has been on for years may not come back on with a power cycle. I've seen it many times. It is something you don't want to do unless you've run out of alternatives (for long term constantly powered equipment). In addition you want to do as much trouble shooting as you can before power cycling to try to determine the cause. Otherwise if it works when repowered it will happen again because the root problem wasn't fixed.

Those root problems can sure be difficult to pin down. There was an intermittent problem that affected Spacelab telemetry data processed in the POCC (Payload Operations Control Center). And it occurred over several missions before it was found. Space Shuttle equipment was split equally between 2 power busses, A and B. It was discovered that the POCC was powered by the C bus. It was discovered that an elevator was on the C bus and was causing the problem via the power bus. Shortly later the POCC was moved to Huntsville.
Title: Re: Switching it off and on again Works: NASA reveals reset fixed Hubble Space telescope's main came
Post by: bigheadfred on January 27, 2019, 05:42:19 am
We have a Brandt edgebander at work. It uses proprietary software. There is a prescribed shutdown procedure. You have to let it do a backup or risk losing its data. They recommend buying a $2,000 surge protector for it. People have just switched it off and killed it. When that happens we have to run a phone line to it so Brandt, in Germany can reload it. When they get around to it. Which can be several days of downtime. Only a licensed tech can access it for simpler error codes.

Sometimes it gets stuck in stupid and your only option is to turn it off and then back on. Take the chance.  It is usually worth the risk. But not as an everyday procedure.