[bug] error importing into newly created project via paste from excel

May 21, 2007 11:59am

David Pattinson 76 post(s) #38

Got the following after attempting to import from excel paste:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /dp_test_import/cards_import/accept.

Reason: Error reading from remote server

Apache/2.2.4 (Fedora) Server at mingle03.thoughtworks.com Port 443

May 21, 2007 12:12pm

David Pattinson 76 post(s) #40

Actually looks like the server is no longer accepting posts at all – I tried editing the homepage of my project and it’s timing out on the post. The above may just be a symptom of an outage.

May 21, 2007 9:20pm

JoeZ 4 post(s) #41

I also had a hard time importing into an existing project. I thought maybe it was because I tried to import too much, but maybe that is not the case.

I tried to import approx 400 rows with about 17 columns each.

May 21, 2007 11:26pm

Julian 10 post(s) #44

if you have trouble importing a spreadsheet, can you forward it to

Jen Marley
Bonno Choi
Julian Boot

and we’ll try and debug what’s failing. 400 cards should be fine – its more likely some data nastiness.

I have a question for people who are having trouble importing: If one or more stories cannot be imported, would you expect the ones that could be imported to be imported? Or would you want the system left in the state it was before you tried to import?

May 22, 2007 4:31am

Jon Tirsen 25 post(s) #47

Performing a POST sets up a transaction so there could be an issue with the database. Is this problem still there?

May 22, 2007 6:45am

Jon Tirsen 25 post(s) #48

We’ve found a number of problems with the import that we are correcting right now. Please try again in a couple of days to see if the problems still persist.

May 22, 2007 1:08pm

JoeZ 4 post(s) #49

Answer to your question above: Yes, I would prefer that you be able to import the ones that are Ok and then you have to go and fix the other ones. But I would probably need a little more functionality in order to be ok with that – like to be able to see a list of which ones failed so that I don’t have to go and figure it out for myself. I could also see the need to choose to cancel the whole import if there are validation errors.

May 23, 2007 12:48am

Julian 10 post(s) #51

joez: I agree, if we go down this path, we’ll need to provide some good feedback on what worked and what was left out, etc. I think this would be a good refinement in a future release.

May 23, 2007 12:55am

David Pattinson 76 post(s) #52

My problem seemed to be with the server, once I’d observed the post error – I got the same error when attempting to edit the hompage of my project.

May 23, 2007 12:58am

David Pattinson 76 post(s) #53

Oh – and I tried importing a from a file created by pasting an export of the current sandbox project into excel, so I’m pretty sure it wasn’t the dataset that was a problem.

May 23, 2007 1:06am

David Pattinson 76 post(s) #54

Re: “I have a question for people who are having trouble importing: If one or more stories cannot be imported, would you expect the ones that could be imported to be imported? Or would you want the system left in the state it was before you tried to import?”

Generally you want what can be imported to go through, with a list of the records that failed. However if more than a specified number of records fails, you generally want to abort the batch to avoid waiting around for nothing. This is probably more important if it becomes possible for people to add large numbers of items to existing projects, in case they miss a column or repeat some identifiers.

You could do with a first pass validation of the initial data row and a fail fast on stuff like “number column must actually be a number”. The first time I tried importing I waited for quite a while to find out that UCxxx was not a valid number.

May 23, 2007 4:38am

Jon Tirsen 25 post(s) #59

The biggest problem with the import right now is that it’s too slow. We denormalize and cache quite a lot of data internally to speed up other operations, this caching is relatively fast for single-item updates but unfortunately it’s gotten out of hand for bulk updates. We’re now working on an infrastructure where these denormalizations will happen asynchronously from the actual bulk import.

This should hopefully make the import so fast that we won’t have to be concerned about these other issues. Let’s reevaluate this at that stage.

May 23, 2007 10:53am

David Pattinson 76 post(s) #63

Jon,

Tried this one again with Julian last night. It looked like only some of the rows actually made it to the database before the proxy error. I’m guessing that means that the batch is not transactional, although the individual rows might be?

May 24, 2007 9:20am

Badri Janaki... Administrator 25 post(s) #89

Dave – I tried to recreate the issue with the sandbox data on mingle03. I was able to import the whole spreadsheet without much issue in just under a minute. The data of DP testing takes a long while – and that is going to get faster with the change that Jon mentions above. Thanks a lot for your comments – and I hope you can test out the sandbox data again soon.

May 29, 2007 2:16pm

David Pattinson 76 post(s) #92

Badri – I think the original problem with the sandbox data may have been a red herring as I tried doing that after I’d already got the server error from attempting to import the larger spreadsheet. I may try doing it again later this week.

[bug] error importing into newly created project via paste from excel#19

Voices

May 21, 2007 11:59am David Pattinson 76 post(s) #38	Got the following after attempting to import from excel paste: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request POST /dp_test_import/cards_import/accept. Reason: Error reading from remote server Apache/2.2.4 (Fedora) Server at mingle03.thoughtworks.com Port 443

May 21, 2007 12:12pm David Pattinson 76 post(s) #40	Actually looks like the server is no longer accepting posts at all – I tried editing the homepage of my project and it’s timing out on the post. The above may just be a symptom of an outage.

May 21, 2007 9:20pm JoeZ 4 post(s) #41	I also had a hard time importing into an existing project. I thought maybe it was because I tried to import too much, but maybe that is not the case. I tried to import approx 400 rows with about 17 columns each.

May 21, 2007 11:26pm Julian 10 post(s) #44	if you have trouble importing a spreadsheet, can you forward it to Jen Marley Bonno Choi Julian Boot and we’ll try and debug what’s failing. 400 cards should be fine – its more likely some data nastiness. I have a question for people who are having trouble importing: If one or more stories cannot be imported, would you expect the ones that could be imported to be imported? Or would you want the system left in the state it was before you tried to import?

May 22, 2007 4:31am Jon Tirsen 25 post(s) #47	Performing a POST sets up a transaction so there could be an issue with the database. Is this problem still there?

May 22, 2007 6:45am Jon Tirsen 25 post(s) #48	We’ve found a number of problems with the import that we are correcting right now. Please try again in a couple of days to see if the problems still persist.

May 22, 2007 1:08pm JoeZ 4 post(s) #49	Answer to your question above: Yes, I would prefer that you be able to import the ones that are Ok and then you have to go and fix the other ones. But I would probably need a little more functionality in order to be ok with that – like to be able to see a list of which ones failed so that I don’t have to go and figure it out for myself. I could also see the need to choose to cancel the whole import if there are validation errors.

May 23, 2007 12:48am Julian 10 post(s) #51	joez: I agree, if we go down this path, we’ll need to provide some good feedback on what worked and what was left out, etc. I think this would be a good refinement in a future release.

May 23, 2007 12:55am David Pattinson 76 post(s) #52	My problem seemed to be with the server, once I’d observed the post error – I got the same error when attempting to edit the hompage of my project.

May 23, 2007 12:58am David Pattinson 76 post(s) #53	Oh – and I tried importing a from a file created by pasting an export of the current sandbox project into excel, so I’m pretty sure it wasn’t the dataset that was a problem.

May 23, 2007 1:06am David Pattinson 76 post(s) #54	Re: “I have a question for people who are having trouble importing: If one or more stories cannot be imported, would you expect the ones that could be imported to be imported? Or would you want the system left in the state it was before you tried to import?” Generally you want what can be imported to go through, with a list of the records that failed. However if more than a specified number of records fails, you generally want to abort the batch to avoid waiting around for nothing. This is probably more important if it becomes possible for people to add large numbers of items to existing projects, in case they miss a column or repeat some identifiers. You could do with a first pass validation of the initial data row and a fail fast on stuff like “number column must actually be a number”. The first time I tried importing I waited for quite a while to find out that UCxxx was not a valid number.

May 23, 2007 4:38am Jon Tirsen 25 post(s) #59	The biggest problem with the import right now is that it’s too slow. We denormalize and cache quite a lot of data internally to speed up other operations, this caching is relatively fast for single-item updates but unfortunately it’s gotten out of hand for bulk updates. We’re now working on an infrastructure where these denormalizations will happen asynchronously from the actual bulk import. This should hopefully make the import so fast that we won’t have to be concerned about these other issues. Let’s reevaluate this at that stage.

May 23, 2007 10:53am David Pattinson 76 post(s) #63	Jon, Tried this one again with Julian last night. It looked like only some of the rows actually made it to the database before the proxy error. I’m guessing that means that the batch is not transactional, although the individual rows might be?

May 24, 2007 9:20am Badri Janaki... Administrator 25 post(s) #89	Dave – I tried to recreate the issue with the sandbox data on mingle03. I was able to import the whole spreadsheet without much issue in just under a minute. The data of DP testing takes a long while – and that is going to get faster with the change that Jon mentions above. Thanks a lot for your comments – and I hope you can test out the sandbox data again soon.

May 29, 2007 2:16pm David Pattinson 76 post(s) #92	Badri – I think the original problem with the sandbox data may have been a red herring as I tried doing that after I’d already got the server error from attempting to import the larger spreadsheet. I may try doing it again later this week.